Skip to content

Commit

Permalink
Add tool compression and encryption support
Browse files Browse the repository at this point in the history 
Updated a number of tools to provide support for dealing with
compressed and/or encrypted data (assuming that the compressed data
is compressed with the GZIP algorithm, and that encrypted data is
written with the PassphraseEncryptedOutputStream).  This includes:

* If ldapsearch output is to be written to one or more LDIF files,
  it is now possible to compress and/or encrypt that output.  The
  --compressOutput argument can be used to indicate that the output
  should be compressed, and the --encryptOutput argument can be
  used to indicate that the output should be encrypted.  If the
  --encryptionPassphraseFile argument is provided, the encryption
  passphrase will be read from that file; otherwise, it will be
  interactively requested from the user.

* If ldapmodify is to read the changes to apply from one or more
  LDIF files, it can automatically detect whether the files are
  encrypted and/or compressed and will handle them accordingly.
  If the --encryptionPassphraseFile argument is provided, the
  encryption passphrase will be read from that file; otherwise, it
  will be interactively requested from the user.

* The split-ldif and transform-ldif tools can now automatically
  detect whether the input LDIF files are compressed and/or
  encrypted, and they can now be directed to encrypt their output.
  The --sourceCompressed argument is still supported for the purpose
  of backward compatibility, but it is no longer required.  The
  --encryptTarget argument can be used to indicate that the output
  should be encrypted.  If the --encryptionPassphraseFile argument
  is provided, then the passphrase will be read from the specified
  file; otherwise, it will be interactively requested from the user.

* The validate-ldif tool can now automatically detect whether the
  input LDIF file is compressed and/or encrypted.  The
  --isCompressed argument is still supported for the purpose of
  backward compatibility, but it is no longer required.  If the
  --encryptionPassphraseFile argument is provided, then the
  passphrase will read from the specified file; otherwise, it will
  be interactively requested from the user.

* The summarize-access-log tool can now automatically detect whether
  the input log files are compressed and/or encrypted.  The
  --isCompressed argument is still supported for the purpose of
  backward compatibility, but it is no longer required.  If the
  --encryptionPassphraseFile argument is provided, then the
  passphrase will read from the specified file; otherwise, it will
  be interactively requested from the user.
  • Loading branch information
@dirmgr
dirmgr committed Mar 7, 2018
1 parent f114592 commit cdb139e
Show file tree
Hide file tree
Showing 22 changed files with 4,240 additions and 264 deletions.
8 changes: 8 additions & 0 deletions docs/release-notes.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,14 @@ <h3>Version 4.0.5</h3>
<br><br>
</li>

<li>
Updated the <tt>ldapsearch</tt>, <tt>ldapmodify</tt>, <tt>split-ldif</tt>,
<tt>transform-ldif</tt>, and <tt>validate-ldif</tt> tools to add support for
encrypted LDIF files. The passphrase used to generate the encryption key can be
provided interactively or read from a file.
<br><br>
</li>

<li>
Updated the <tt>summarize-access-log</tt> tool to add support for encrypted log
files. The passphrase used to generate the encryption key can be provided
Expand Down
69 changes: 66 additions & 3 deletions messages/unboundid-ldapsdk-tools.properties
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1263,6 +1263,10 @@ INFO_LDAPMODIFY_ARG_DESCRIPTION_LDIF_FILE=The path to the LDIF file \
changes will be read from standard input. If this argument is provided \
multiple times to supply multiple LDIF files, then they will be processed \
in the order they were provided on the command line.
INFO_LDAPMODIFY_ARG_DESCRIPTION_ENCRYPTION_PW_FILE=The path to a file that \
contains the passphrase used to generate the key used to encrypt the LDIF \
data. If the data is encrypted and this argument is not provided, then the \
tool will interactively prompt for the correct password.
INFO_LDAPMODIFY_PLACEHOLDER_CHARSET={charset}
INFO_LDAPMODIFY_ARG_DESCRIPTION_CHARACTER_SET=The character set for the \
LDIF data to be read. If this argument is not provided, a default encoding \
Expand Down Expand Up @@ -1369,6 +1373,13 @@ ERR_LDAPMODIFY_CANNOT_CREATE_CONNECTION_POOL=An error occurred while \
server: {0}
INFO_LDAPMODIFY_CONNECTION_ESTABLISHED=Successfully connected to {0}.
INFO_LDAPMODIFY_STARTED_TXN=Successfully started a transaction with ID {0}
ERR_LDAPMODIFY_ENCRYPTION_PW_FILE_EMPTY=Encryption passphrase file ''{0}'' \
is empty.
ERR_LDAPMODIFY_ENCRYPTION_PW_FILE_MULTIPLE_LINES=Encryption passphrase file \
''{0}''has multiple lines. The file must contain exactly one line, which \
is comprised entirely of the encryption passphrase.
ERR_LDAPMODIFY_ENCRYPTION_PW_FILE_READ_ERROR=An error occurred while trying \
to read the encryption passphrase from file ''{0}'': {1}
ERR_LDAPMODIFY_CANNOT_START_TXN=An error occurred while trying to start an \
LDAP transaction: {0}
ERR_LDAPMODIFY_CANNOT_CREATE_LDIF_READER=An error occurred while attempting \
Expand Down Expand Up @@ -1719,6 +1730,23 @@ INFO_LDAPSEARCH_ARG_DESCRIPTION_OUTPUT_FORMAT=Specifies the format that \
INFO_LDAPSEARCH_ARG_DESCRIPTION_OUTPUT_FILE=Specifies the path to the file to \
which search results should be written. If this argument is not provided \
then results will be written to standard output.
INFO_LDAPSEARCH_ARG_DESCRIPTION_COMPRESS_OUTPUT=Indicates that the output \
should be gzip-compressed. This can only be used if the --outputFile \
argument is provided and the --teeResultsToStandardOut argument is not \
provided.
INFO_LDAPSEARCH_ARG_DESCRIPTION_ENCRYPT_OUTPUT=Indicates that the output \
should be encrypted with a key generated from a provided password. This \
can only be used if the --outputFile argument is provided and the \
--teeResultsToStandardOut argument is not provided. If the \
--encryptionPassphraseFile argument is provided, then that file will be used \
to specify the encryption passphrase; otherwise, the passphrase will be \
interactively requested.
INFO_LDAPSEARCH_ARG_DESCRIPTION_ENCRYPTION_PW_FILE=The path to a file that \
specifies the passphrase to use to encrypt the output. This can only be \
provided if the --encryptOutput argument is given, but if that argument is \
given and no passphrase file is specified, then the passphrase will be \
interactively requested. If a file is specified, then that file must \
exist and must contain exactly one line comprised entirely of the passphrase.
INFO_LDAPSEARCH_ARG_DESCRIPTION_ASSERTION_FILTER=A filter that will be used \
in conjunction with the LDAP assertion request control (as described in RFC \
4528) to indicate that the server should only process searches in which the \
Expand Down Expand Up @@ -1988,7 +2016,7 @@ ERR_LDAPSEARCH_VLV_INVALID_VALUE=The value provided for the ''{0}'' argument \
is invalid. The value must either be a colon-delimited list of four \
integers (representing the before count; after count; offset index; and \
expected result set size, with zero used if the expected result set size is \
not knonw), or a colon-delimited list of two integers and one string \
not known), or a colon-delimited list of two integers and one string \
(representing the before count, after count, and the primary sort attribute \
value that should be used as the start of the result set).
ERR_LDAPSEARCH_JOIN_RULE_INVALID_VALUE=The value provided for the ''{0}'' \
Expand Down Expand Up @@ -2017,8 +2045,8 @@ ERR_LDAPSEARCH_MOVE_SUBTREE_MISMATCH=The --moveSubtreeFrom argument must be \
ERR_LDAPSEARCH_OUTPUT_FORMAT_NOT_SUPPORTED_WITH_URLS=The ''{0}'' output \
format cannot be used in conjunction with the ''{1}'' argument.
ERR_LDAPSEARCH_OUTPUT_FORMAT_REQUIRES_REQUESTED_ATTR_ARG=The ''{0}'' output \
format requires that the set of requested arguments be identified with the \
''{1}'' argument rather than as unnamed trailing arguments.
format requires that the set of requested arguments be identified with the \
''{1}'' argument rather than as unnamed trailing arguments.
ERR_LDAPSEARCH_CANNOT_OPEN_OUTPUT_FILE=Unable to open output file ''{0}'' \
for writing: {1}
ERR_LDAPSEARCH_CANNOT_CREATE_CONNECTION_POOL=An error occurred while \
Expand Down Expand Up @@ -2101,6 +2129,18 @@ INFO_SPLIT_LDIF_GLOBAL_ARG_DESC_TARGET_LDIF_BASE=The path and base name to \
source LDIF path will be used as the base name.
INFO_SPLIT_LDIF_GLOBAL_ARG_DESC_COMPRESS_TARGET=Indicates that the target \
LDIF files should be gzip-compressed.
INFO_SPLIT_LDIF_GLOBAL_ARG_DESC_ENCRYPT_TARGET=Indicates that the target LDIF \
LDIF files should be encrypted with a key generated from a provided \
passphrase. If the --encryptionPassphraseFile argument is provided, then \
the passphrase will be read from the specified file. Otherwise, it will be \
interactively requested from the user.
INFO_SPLIT_LDIF_GLOBAL_ARG_DESC_ENCRYPT_PW_FILE=The path to the file \
containing the passphrase to use to generate the encryption key. This \
passphrase will be used to decrypt the input (if it is encrypted) and to \
encrypt the output (if it is to be encrypted). If this is not provided and either the input or output is encrypted, then the passphrase will be \
interactively requested. If it is provided, then the specified file must \
exist and must contain exactly one line that is comprised entirely of the \
encryption passphrase.
INFO_SPLIT_LDIF_GLOBAL_ARG_DESC_SPLIT_BASE_DN=The base DN below which entries \
should be split. The entry with this DN will appear in all sets, but each \
entry below this base DN will appear in exactly one set. This must be \
Expand Down Expand Up @@ -2302,4 +2342,27 @@ ERR_TOOL_LOGGER_ERROR_OPENING_LOG_FILE=An error occurred while trying to open \
log file ''{0}'' for writing an invocation log message: {1}
ERR_TOOL_LOGGER_UNABLE_TO_ACQUIRE_FILE_LOCK=Unable to acquire an exclusive \
lock on tool invocation log file ''{0}'' after {1,number,0} attempts.
ERR_TOOL_UTILS_ENCRYPTION_PW_FILE_MISSING=Encryption passphrase file ''{0}'' \
does not exist.
ERR_TOOL_UTILS_ENCRYPTION_PW_FILE_NOT_FILE=Encryption passphrase file ''{0}'' \
exists but is not a file.
ERR_TOOL_UTILS_ENCRYPTION_PW_FILE_EMPTY=Encryption passphrase file ''{0}'' is \
empty. The file must consist of exactly one line that is comprised \
entirely of the encryption passphrase.
ERR_TOOL_UTILS_ENCRYPTION_PW_FILE_MULTIPLE_LINES=Encryption passphrase file \
''{0}'' contains multiple lines. The file must consist of exactly one line \
that is comprised entirely of the encryption passphrase.
ERR_TOOL_UTILS_ENCRYPTION_PW_FILE_READ_ERROR=An error occurred while \
attempting to read the encryption passphrase from file ''{0}'': {1}
INFO_TOOL_UTILS_ENCRYPTION_PW_PROMPT=Enter the encryption passphrase:
ERR_TOOL_UTILS_ENCRYPTION_PW_EMPTY=ERROR: The encryption passphrase must not \
be empty.
INFO_TOOL_UTILS_ENCRYPTION_PW_CONFIRM=Confirm the encryption passphrase:
ERR_TOOL_UTILS_ENCRYPTION_PW_MISMATCH=ERROR: The passphrases do not match.
INFO_TOOL_UTILS_ENCRYPTED_LDIF_FILE_PW_PROMPT=LDIF file ''{0}'' is \
encrypted. Please enter the passphrase required to decrypt it:
ERR_TOOL_UTILS_ENCRYPTED_LDIF_FILE_WRONG_PW=The provided passphrase is \
incorrect.
ERR_TOOL_UTILS_ENCRYPTED_LDIF_FILE_CANNOT_DECRYPT=LDIF file ''{0}'' is \
encrypted, but an error occurred while attempting to decrypt it: {1}

13 changes: 12 additions & 1 deletion messages/unboundid-ldapsdk-transformations.properties
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,17 @@ INFO_TRANSFORM_LDIF_ARG_DESC_SOURCE_COMPRESSED=Indicates that the source LDIF \
files are gzip-compressed.
INFO_TRANSFORM_LDIF_ARG_DESC_COMPRESS_TARGET=Indicates that the target LDIF \
file should be gzip-compressed.
INFO_TRANSFORM_LDIF_ARG_DESC_ENCRYPT_TARGET=Indicates that the target LDIF \
file should be encrypted with a key generated from a provided passphrase. \
If the --encryptionPassphraseFile argument is provided, then the passphrase \
will be read from that file; otherwise, it will be interactively requested.
INFO_TRANSFORM_LDIF_ARG_DESC_ENCRYPTION_PW_FILE=The path to a file that \
contains the passphrase that should be used to generate the encryption \
key, and also to decrypt the input if it happens to be encrypted. If the \
--encryptTarget argument is provided and no passphrase file is given, then \
the passphrase will be interactively requested. If an encryption \
passphrase file is specified, then it must contain exactly one line, and \
that line must be comprised entirely of the passphrase.
INFO_TRANSFORM_LDIF_ARG_GROUP_SCRAMBLE=Arguments for Scrambling Attribute \
Values
INFO_TRANSFORM_LDIF_PLACEHOLDER_ATTR_NAME={attributeName}
Expand Down Expand Up @@ -359,6 +370,6 @@ INFO_TRANSFORM_LDIF_EXAMPLE_ADD=Transform the data contained in file \
'added-organization.ldif' file.
INFO_TRANSFORM_LDIF_EXAMPLE_REBASE=Transform the data contained in file \
'input.ldif' to move all entries at or below 'o=example.com' so that they \
will instead be below 'dc=exmaple,dc=com'. The output will be written to \
will instead be below 'dc=example,dc=com'. The output will be written to \
the 'rebased.ldif' file.

2 changes: 2 additions & 0 deletions messages/unboundid-ldapsdk-util.properties
View file
Original file line number Diff line number Diff line change
Expand Up @@ -813,3 +813,5 @@ ERR_CLOSEABLE_RW_LOCK_TRY_LOCK_WRITE_TIMEOUT=Unable to acquire the closeable \
write lock with a timeout of {0}.
ERR_CLOSEABLE_RW_LOCK_TRY_LOCK_READ_TIMEOUT=Unable to acquire the closeable \
read lock with a timeout of {0}.
ERR_CANNOT_GET_ENCRYPTION_PASSPHRASE=Unable to interactively read the \
encryption passphrase from the user: {0}
88 changes: 84 additions & 4 deletions src/com/unboundid/ldap/sdk/examples/ValidateLDIF.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,13 @@
import com.unboundid.ldap.sdk.Version;
import com.unboundid.ldap.sdk.schema.Schema;
import com.unboundid.ldap.sdk.schema.EntryValidator;
import com.unboundid.ldap.sdk.unboundidds.tools.ToolUtils;
import com.unboundid.ldif.DuplicateValueBehavior;
import com.unboundid.ldif.LDIFException;
import com.unboundid.ldif.LDIFReader;
import com.unboundid.ldif.LDIFReaderEntryTranslator;
import com.unboundid.ldif.LDIFWriter;
import com.unboundid.util.Debug;
import com.unboundid.util.LDAPCommandLineTool;
import com.unboundid.util.ThreadSafety;
import com.unboundid.util.ThreadSafetyLevel;
Expand Down Expand Up @@ -172,6 +174,7 @@ public final class ValidateLDIF
private FileArgument schemaDirectory;
private FileArgument ldifFile;
private FileArgument rejectFile;
private FileArgument encryptionPassphraseFile;
private IntegerArgument numThreads;
private StringArgument ignoreSyntaxViolationsForAttribute;

Expand Down Expand Up @@ -409,18 +412,46 @@ protected boolean includeAlternateLongIdentifiers()
public void addNonLDAPArguments(final ArgumentParser parser)
throws ArgumentException
{
String description = "The path to the LDIF file to process.";
String description = "The path to the LDIF file to process. The tool " +
"will automatically attempt to detect whether the file is " +
"encrypted or compressed.";
ldifFile = new FileArgument('f', "ldifFile", true, 1, "{path}", description,
true, true, true, false);
ldifFile.addLongIdentifier("ldif-file", true);
parser.addArgument(ldifFile);


// Add an argument that makes it possible to read a compressed LDIF file.
// Note that this argument is no longer needed for dealing with compressed
// files, since the tool will automatically detect whether a file is
// compressed. However, the argument is still provided for the purpose of
// backward compatibility.
description = "Indicates that the specified LDIF file is compressed " +
"using gzip compression.";
isCompressed = new BooleanArgument('c', "isCompressed", description);
isCompressed.addLongIdentifier("is-compressed", true);
isCompressed.setHidden(true);
parser.addArgument(isCompressed);


// Add an argument that indicates that the tool should read the encryption
// passphrase from a file.
description = "Indicates that the specified LDIF file is encrypted and " +
"that the encryption passphrase is contained in the specified " +
"file. If the LDIF data is encrypted and this argument is not " +
"provided, then the tool will interactively prompt for the " +
"encryption passphrase.";
encryptionPassphraseFile = new FileArgument(null,
"encryptionPassphraseFile", false, 1, null, description, true, true,
true, false);
encryptionPassphraseFile.addLongIdentifier("encryption-passphrase-file",
true);
encryptionPassphraseFile.addLongIdentifier("encryptionPasswordFile", true);
encryptionPassphraseFile.addLongIdentifier("encryption-password-file",
true);
parser.addArgument(encryptionPassphraseFile);


description = "The path to the file to which rejected entries should be " +
"written.";
rejectFile = new FileArgument('R', "rejectFile", false, 1, "{path}",
Expand Down Expand Up @@ -634,6 +665,7 @@ public ResultCode doToolProcessing()
}
catch (final Exception e)
{
Debug.debugException(e);
err("Unable to read schema from files in directory " +
schemaDir.getAbsolutePath() + ": " + getExceptionMessage(e));
return ResultCode.LOCAL_ERROR;
Expand All @@ -649,13 +681,32 @@ public ResultCode doToolProcessing()
}
catch (final LDAPException le)
{
Debug.debugException(le);
err("Unable to connect to the directory server and read the schema: ",
le.getMessage());
return le.getResultCode();
}
}


// Get the encryption passphrase, if it was provided.
String encryptionPassphrase = null;
if (encryptionPassphraseFile.isPresent())
{
try
{
encryptionPassphrase = ToolUtils.readEncryptionPassphraseFromFile(
encryptionPassphraseFile.getValue());
}
catch (final LDAPException e)
{
Debug.debugException(e);
err(e.getMessage());
return e.getResultCode();
}
}


// Create the entry validator and initialize its configuration.
entryValidator = new EntryValidator(schema);
entryValidator.setCheckAttributeSyntax(!ignoreAttributeSyntax.isPresent());
Expand Down Expand Up @@ -693,14 +744,29 @@ public ResultCode doToolProcessing()
try
{
InputStream inputStream = new FileInputStream(ldifFile.getValue());

inputStream = ToolUtils.getPossiblyPassphraseEncryptedInputStream(
inputStream, encryptionPassphrase, false,
"LDIF file '" + ldifFile.getValue().getPath() +
"' is encrypted. Please enter the encryption passphrase:",
"ERROR: The provided passphrase was incorrect.",
getOut(), getErr()).getFirst();

if (isCompressed.isPresent())
{
inputStream = new GZIPInputStream(inputStream);
}
else
{
inputStream =
ToolUtils.getPossiblyGZIPCompressedInputStream(inputStream);
}

ldifReader = new LDIFReader(inputStream, numThreads.getValue(), this);
}
catch (final Exception e)
{
Debug.debugException(e);
err("Unable to open the LDIF reader: ", getExceptionMessage(e));
return ResultCode.LOCAL_ERROR;
}
Expand Down Expand Up @@ -728,6 +794,7 @@ public ResultCode doToolProcessing()
}
catch (final Exception e)
{
Debug.debugException(e);
err("Unable to create the reject writer: ", getExceptionMessage(e));
return ResultCode.LOCAL_ERROR;
}
Expand All @@ -751,6 +818,7 @@ public ResultCode doToolProcessing()
}
catch (final LDIFException le)
{
Debug.debugException(le);
malformedEntries.incrementAndGet();

if (resultCode == ResultCode.SUCCESS)
Expand Down Expand Up @@ -780,6 +848,7 @@ public ResultCode doToolProcessing()
}
catch (final IOException ioe)
{
Debug.debugException(ioe);
err("Unable to write to the reject file:",
getExceptionMessage(ioe));
err("LDIF parse failure that triggered the rejection: ",
Expand All @@ -790,6 +859,7 @@ public ResultCode doToolProcessing()
}
catch (final IOException ioe)
{
Debug.debugException(ioe);

if (rejectWriter != null)
{
Expand All @@ -803,6 +873,7 @@ public ResultCode doToolProcessing()
}
catch (final Exception ex)
{
Debug.debugException(ex);
err("I/O error reading from LDIF:", getExceptionMessage(ioe));
return ResultCode.LOCAL_ERROR;
}
Expand Down Expand Up @@ -844,7 +915,10 @@ public ResultCode doToolProcessing()
{
ldifReader.close();
}
catch (final Exception e) {}
catch (final Exception e)
{
Debug.debugException(e);
}

try
{
Expand All @@ -853,7 +927,10 @@ public ResultCode doToolProcessing()
rejectWriter.close();
}
}
catch (final Exception e) {}
catch (final Exception e)
{
Debug.debugException(e);
}
}
}

Expand Down Expand Up @@ -885,7 +962,10 @@ public Entry translate(final Entry entry, final long firstLineNumber)
{
rejectWriter.writeEntry(entry, listToString(invalidReasons));
}
catch (final IOException ioe) {}
catch (final IOException ioe)
{
Debug.debugException(ioe);
}
}
}
}
Expand Down
Loading

0 comments on commit cdb139e

Please sign in to comment.