Class ScratchFile
- All Implemented Interfaces:
Closeable
,AutoCloseable
RandomAccess
buffers each having its set of pages (implemented by
ScratchFileBuffer
). A buffer is created calling createBuffer()
.
Pages can be stored in main memory or in a temporary file. A mixed mode is supported storing a certain amount of pages in memory and only the additional ones in temporary file (defined by maximum main memory to be used).
Pages can be marked as 'free' in order to re-use them. For in-memory pages this will release the used memory while for pages in temporary file this simply marks the area as free to re-use.
If a temporary file was created (done with the first page to be stored
in temporary file) it is deleted when close()
is called.
Using this class for RandomAccess
buffers allows for a direct control
on the maximum memory usage and allows processing large files for which we
otherwise would get an OutOfMemoryError
in case of using RandomAccessBuffer
.
This base class for providing pages is thread safe (the buffer implementations are not).
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final List<ScratchFileBuffer>
private static final int
number of pages by which we enlarge the scratch file (reduce I/O-operations)private File
scratch file; only to be accessed under synchronization ofioLock
private final BitSet
private static final int
in case of unrestricted main memory usage this is the initial number of pagesinMemoryPages
is setup forprivate final int
private byte[][]
holds pointers to in-memory page content; will be initialized once in case of restricted main memory, otherwise it is enlarged as needed and first initialized to a size ofINIT_UNRESTRICTED_MAINMEM_PAGECOUNT
private final Object
private boolean
private static final org.apache.commons.logging.Log
private final boolean
private final int
private static final int
private int
private RandomAccessFile
random access to scratch file; only to be accessed under synchronization ofioLock
private final File
private final boolean
-
Constructor Summary
ConstructorsConstructorDescriptionScratchFile
(File scratchFileDirectory) Initializes page handler.ScratchFile
(MemoryUsageSetting memUsageSetting) Initializes page handler. -
Method Summary
Modifier and TypeMethodDescription(package private) void
Checks if this page handler has already been closed.void
close()
Closes and deletes the temporary file.Creates a new buffer using this page handler.createBuffer
(InputStream input) Creates a new buffer using this page handler and initializes it with the data read from provided input stream (input stream is copied to buffer).private void
enlarge()
This will provide new free pages by either enlarging the scratch file by a number of pages defined byENLARGE_PAGE_COUNT
- in case scratch file usage is allowed - or increase theinMemoryPages
array in case main memory was not restricted.static ScratchFile
Getter for an instance using only unrestricted main memory for buffering (same asnew ScratchFile(MemoryUsageSetting.setupMainMemoryOnly())
).(package private) int
Returns a new free page, either from free page pool or by enlarging scratch file (may be created).(package private) int
Returns byte size of a page.(package private) void
markPagesAsFree
(int[] pageIndexes, int off, int count) Allows a buffer which is cleared/closed to release its pages to be re-used.(package private) byte[]
readPage
(int pageIdx) Reads the page with specified index.(package private) void
removeBuffer
(ScratchFileBuffer buffer) (package private) void
writePage
(int pageIdx, byte[] page) Writes updated page.
-
Field Details
-
LOG
private static final org.apache.commons.logging.Log LOG -
ENLARGE_PAGE_COUNT
private static final int ENLARGE_PAGE_COUNTnumber of pages by which we enlarge the scratch file (reduce I/O-operations)- See Also:
-
INIT_UNRESTRICTED_MAINMEM_PAGECOUNT
private static final int INIT_UNRESTRICTED_MAINMEM_PAGECOUNTin case of unrestricted main memory usage this is the initial number of pagesinMemoryPages
is setup for- See Also:
-
PAGE_SIZE
private static final int PAGE_SIZE- See Also:
-
ioLock
-
scratchFileDirectory
-
file
scratch file; only to be accessed under synchronization ofioLock
-
raf
random access to scratch file; only to be accessed under synchronization ofioLock
-
pageCount
private volatile int pageCount -
freePages
-
inMemoryPages
private volatile byte[][] inMemoryPagesholds pointers to in-memory page content; will be initialized once in case of restricted main memory, otherwise it is enlarged as needed and first initialized to a size ofINIT_UNRESTRICTED_MAINMEM_PAGECOUNT
-
inMemoryMaxPageCount
private final int inMemoryMaxPageCount -
maxPageCount
private final int maxPageCount -
useScratchFile
private final boolean useScratchFile -
maxMainMemoryIsRestricted
private final boolean maxMainMemoryIsRestricted -
buffers
-
isClosed
private volatile boolean isClosed
-
-
Constructor Details
-
ScratchFile
Initializes page handler. If ascratchFileDirectory
is supplied, then the scratch file will be created in that directory.All pages will be stored in the scratch file.
- Parameters:
scratchFileDirectory
- The directory in which to create the scratch file ornull
to created it in the default temporary directory.- Throws:
IOException
- If scratch file directory was given but don't exist.
-
ScratchFile
Initializes page handler. If ascratchFileDirectory
is supplied, then the scratch file will be created in that directory.Depending on the size of allowed memory usage a number of pages (memorySize/
PAGE_SIZE
) will be stored in-memory and only additional pages will be written to/read from scratch file.- Parameters:
memUsageSetting
- set how memory/temporary files are used for buffering streams etc.- Throws:
IOException
- If scratch file directory was given but don't exist.
-
-
Method Details
-
getMainMemoryOnlyInstance
Getter for an instance using only unrestricted main memory for buffering (same asnew ScratchFile(MemoryUsageSetting.setupMainMemoryOnly())
).- Returns:
- instance configured to only use main memory with no size restriction
-
getNewPage
Returns a new free page, either from free page pool or by enlarging scratch file (may be created).- Returns:
- index of new page
- Throws:
IOException
-
enlarge
This will provide new free pages by either enlarging the scratch file by a number of pages defined byENLARGE_PAGE_COUNT
- in case scratch file usage is allowed - or increase theinMemoryPages
array in case main memory was not restricted. If neither of both is allowed/the case than free pages count won't be changed. The same is true if no new pages could be added because we reached the maximum ofInteger.MAX_VALUE
pages.If scratch file usage is allowed and scratch file does not exist already it will be created.
Only to be called under synchronization on
freePages
.- Throws:
IOException
-
getPageSize
int getPageSize()Returns byte size of a page.- Returns:
- byte size of a page
-
readPage
Reads the page with specified index.- Parameters:
pageIdx
- index of page to read- Returns:
- byte array of size
PAGE_SIZE
filled with page data read from file - Throws:
IOException
-
writePage
Writes updated page. Page is either kept in-memory if pageIdx <inMemoryMaxPageCount
or is written to scratch file.Provided page byte array must not be re-used for other pages since we store it as is in case of in-memory handling.
- Parameters:
pageIdx
- index of page to writepage
- page to write (length has to be 4096)- Throws:
IOException
- in case page index is out of range or page has wrong length or writing to file failed
-
checkClosed
Checks if this page handler has already been closed. If so, anIOException
is thrown.- Throws:
IOException
- Ifclose()
has already been called.
-
createBuffer
Creates a new buffer using this page handler.- Returns:
- A new buffer.
- Throws:
IOException
- If an error occurred.
-
removeBuffer
-
createBuffer
Creates a new buffer using this page handler and initializes it with the data read from provided input stream (input stream is copied to buffer). The buffer data pointer is reset to point to first byte.- Parameters:
input
- The input stream that is to be copied into the buffer.- Returns:
- A new buffer containing data read from input stream.
- Throws:
IOException
- If an error occurred.
-
markPagesAsFree
void markPagesAsFree(int[] pageIndexes, int off, int count) Allows a buffer which is cleared/closed to release its pages to be re-used.- Parameters:
pageIndexes
- pages indexes of pages to releasecount
- number of page indexes contained in provided array
-
close
Closes and deletes the temporary file. No further interaction with the scratch file or associated buffers can happen after this method is called. It also releases in-memory pages.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
- If there was a problem closing or deleting the temporary file.
-