|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Java .BlendDocumentationVersion 1.3.10 11-Dec-2022 Holger Machens This is the documentation of the Java .Blend generic Blender file import/export toolkit for Java. Blender is an open source 3D graphics modelling and rendering environment which supports import and export of various file formats such as FBX and Collada. Imports and exports always require a mapping between different formats which inevitably involves compromises, resulting in loss of information. Java .Blend aims to fill the gap for Java programmers, who want type-safe access to all the data in a Blender file with read and write capabilities. To achieve that, Java .Blend uses meta data which is contained in each Blender file, and describes Blender's entire core data model. This meta data is fed to Java .Blend's code generator, which generates Java classes for all types in Blender's data model and thereby creates a type-safe API to access raw data in a This document contains a system specification of Java .Blend. It describes its functionality and specifies its data model. To understand the system specification, and to work with Blender files, a basic knowledge of Blender's data model and its file format is necessary. Therefore, this document starts with an introduction to Blender's DNA and its file format. The section thereafter covers the system specification which involves vital information on the type mapping between C structs and types in Blender and Java classes and types in Java .Blend. It also explains main parts of the Java .Blend runtime library (data I/O). More detailed information on parts of the system can be found in the Javadoc documentation (respective the source code documentation). At the end of this document there is a brief getting started guide for API programmers. Main purpose of this section is to point programmers in the right direction. The examples found in the download section of the website are intended to help further down the road. ContentsBlender DNA Blender's core data model is called its DNA. It covers all relevant, persistent data of a running Blender process which will be stored in a file. Data is stored in a Brief Data Model Overview Blender organises all data in so called libraries. A library holds references on the content of one file. A library is either classified as a Main libraries (see Figure above) hold references on the main library elements such as a Further down, main library elements have references between each other to establish a scene graph (see Figure below). Blender File FormatOn its lowest level, a Blender file consists of a header followed by a list of blocks of memory. Each block holds data of a specific type, such as a struct an array of structs or single data elements. One of the blocks contains the type information required to interpret the body of a block. File HeaderThe file header (first 12 bytes of every Blender file) contains low-level information about the file:
As you can see, all the data given in the header is represented in single bytes and therefore independent of byte order and address width of the underlying system architecture. All subsequent data following after the file header may require conversion to the data representation of the runtime environment, present. Block LevelBlocks consist of a header, which describes what type of data is stored in its body, and a body with the actual data. The block header always starts at a file position which is 4 byte aligned. The body always contains data of one specific type, which is either a struct, an array of structs or scalars or even single scalar elements.
Since each block has its original memory address stored, it is possible to restore pointers on data in blocks. But there are cases, where the pointer addresses memory which is not in the file, too. Type Information All type information is stored in the block with code
Blender Address Mapping Since all data of a Blender process is stored without any conversion, all pointers in structs will still reference addresses in the memory of the original process. When Blender loads a file, data will get a different address in the new process and all pointers in that data have to be adjusted to point to the new location. For the most part, blender uses a so-called Please note, that the start address of a block is guaranteed to be unique but the content may or may not overlap the address space of following blocks. As explained, most of the blocks contain data which originates from the heap of the blender process which created the file. In a healthy application, memory areas on heap do not overlap each other. Accordingly, all addresses referring to data inside of any of those areas stored in a block are unique as well. However, there are a few exceptions since not all of the blocks existed in heap memory in the form they are written to the file. One example is a tree structure which is written to the file as a list to get a more compact format. The start address of the block which contains the tree elements is artificial and does not correspond to an actual address on heap. Blender guarantees that the start address is unique, but the address range represented by the data in this block (i.e. the range Blender Versioning and File VersionBlender differentiates between Blender program version (referred to as Blender version) and Blender file version. This section clarifies the meaning and relationship of both version specifiers. On release, both types of version specifiers are comprised of three numbers which have different meanings:
Beyond that, there is also a version cycle specifier, which reflects different stages in the development process, such as alpha, beta, release candidate (rc) and finally release. However, any officially released version (including patch releases) has the version cycle specifier release, which is why it can be ignored here. As initially mentioned, Blender developers have decided to separate versioning of the Blender program code and the Blender file format.
To keep track of the relationship between Blender version and Blender file version, both version numbers will always have the same major and minor number. Thus, a Blender program of version 2.91.x will always create files of file version 2.91.y, where x and y can be different. Blender generally allows to read older version Blender files into a newer version Blender program. This requires Blender to have the code to migrate the DNA stored in an older version file into a new version DNA. To some degree, Blender can also migrate backwards, which means that a Blender program of an older version can read a Blender file of a newer version. However, this works only in a limited version range and this range is specified by two file version: The actual file version and the so-called 'minimal' version. Below is the comment in the code, which states that. Blender achives that by storing the actual layout of each struct and offsets of their properties in each blender file in the block DNA1. /* Blender major and minor version. */ #define BLENDER_VERSION 291 /* Blender patch version for bugfix releases. */ #define BLENDER_VERSION_PATCH 0 /** Blender release cycle stage: alpha/beta/rc/release. */ #define BLENDER_VERSION_CYCLE release /* Blender file format version. */ #define BLENDER_FILE_VERSION BLENDER_VERSION #define BLENDER_FILE_SUBVERSION 10 /* Minimum Blender version that supports reading file written with the current * version. Older Blender versions will test this and show a warning if the file * was written with too new a version. */ #define BLENDER_FILE_MIN_VERSION 290 #define BLENDER_FILE_MIN_SUBVERSION 0 The version specifier given in the file header of a Blender file always refers to the version of the Blender program instance, which created the file. Besides this information, a Blender file contains a block called A full version check in Blender considers these four version specifiers:
Based on this information a system can consider a given file to be compatible with Blender in a version starting from Block Order Blocks stored in a Bender file follow a specific order. The common rule for blocks stored in a Blender file is, that referenced The following list summarises all block order related rules:
ConceptDesign OverviewThis section gives a brief overview of the general concept behind Java .Blend. ModulesJava .Blend basically consists of two modules:
To summarise: Based on the type information (see Type Information) of any reference Blender file, the model generator generates a data model in terms of facades which provide type-safe access to native Blender data. The generated data model in combination with the runtime data I/O module is used at runtime to access data of any Blender file of the same version. Native Blender data is kept in its format in heap memory and accessed through the facades getter and setter methods. Life cycle of native data is implicitly covered by the Java VMs garbage collection. Method At development time, the model generator is used to generate a data model for a particular Blender version, based on any Example java -jar JavaBlend.jar -out "../myproject/src" -p "my.package.blender" -in "any-file.blend" Once the data model is generated, it can be used in combination with the runtime data I/O module to access any Example // // Open file and retrieve main lib // BlenderFile f = new BlenderFile("cube.blend"); MainLib main = new MainLib(f); f.close(); // // retrieve the first mesh's polygons // Mesh mesh = lib.getMesh(); MPoly[] polygons = mesh.getMpoly().toArray(mesh.getTotpoly()); More comprehensive examples involving materials and textures can be found in the download section of the website. Block Level I/O On the lowest level, the runtime I/O package has to deal with blocks and mapping of addresses to offsets in blocks (see package Address Resolution ConceptFor all blocks which contain data originating from heap, address mapping is straight forward: Lookup the block with a start address less or equal to the address which has to be resolved and whose length is large enough to contain the address. Offheap AreasAs explained in Section Blender Address Mapping a .blend file can contain blocks whose address range may or may not overlap the address range of other blocks. Since the data did not originate from its original location on the heap of the blender process which created the file, we call the corresponding memory area Offheap Areas. The general rule for blocks in offheap areas is, that only their start address is guaranteed to be unique. Luckily, only certain types of data are stored in offheap areas. Thus, we can differentiate between on-heap and offheap areas based on the type of data which is referenced (also stored in the block header) and apply the proper address resolution routine. The types of blocks, which originated from offheap areas may change with different versions of blender. Thus, the I/O subsystem contains a lookup table with lists of struct names to be found in offheap areas for different blender versions (see package In case a .blend file contains blocks which overlap each other and whose struct types are not listed in the offheap areas, the block table will throw an exception during initialisation. This exception provides a list of the overlapping blocks and their struct types. This information is needed to fix the issue. To fix an issue with overlapping blocks, you need to adapt the list of types from offheap areas to the given blender version. Simply add a new entry in class map.add(new Entry(280, new String[]{"FileGlobal", "TreeStoreElem"})); To identify the type of data which should be moved to an offheap area, you can use the information given by the OverlappingBlocksException you have received. Try to identify the type which solves the most overlap conflicts at once when moved off heap and add it to the list of your new offheap area entry. Important: Offheap areas cannot contain the type Afterwards, run the code generation again, add the new code to your project and read the file again to test it. Another approach is off course to study the blender source code and identify the actual structure, which causes the issue ;) Class BlenderFile When creating new Blender files, this class is also responsible to initialise the file with essential data, such as header, end block and a block with type information ( When finally writing a created Blender file, this class adds the fundamental blocks Block TableThe component called block table is responsible to map addresses to blocks according to the concept explained in Section Address Resolution). Internally, the block table sorts all blocks by their base address to optimise lookup performance. To handle offheap areas transparently, the block table references a set of block tables, which contain the blocks of those offheap areas. Block ListBlocks in a Blender file follow a specific order, as explained in Section Block Order. Since the block table reorders blocks according to their base address, the block list was added to keep track of the sequential order required in a Blender file. Internally, the block list maintains a linked list of blocks to allow fast insertion or removal of blocks. AllocatorBlocks of one Blender file establish a virtual memory space based on blocks original address and size (cf. Address Mapping). Especially when adding content to an existing or new Blender file, low level I/O has to keep track of allocated and free memory in respect to existing blocks. When inserting new blocks, the low level I/O has to check, where this new block can be placed in this virtual memory space. The allocator is assigned to the block table and is responsible to keep track of the virtual memory space of the associated Blender files in terms of free and allocated memory regions. The allocator is not responsible to actually allocate memory in the JVM. This is a task of the block table. The allocator just finds free address space for a new block or removes address space from its internal list of allocated regions. Meta Data RepresentationMeta data (type information) of a Blender file is available in three different formats:StructDNA ,
BlendModel and
CMetaModel .
StructDNAThe classStructDNA , which is an exact replica of the file content in block
"DNA1" (see Section
Type Information), provides no higher functionalities, such as lookup of types.
BlendModelThe classBlendModel basically contains the same information as
StructDNA , but provides lookup capabilities.
CMetaModelThe classCMetaModel , provides an enriched meta model which contains classification of types (such as pointer, array, scalar, struct) and detailed break downs of types into their base types (see class documentation for more information).
Type MappingThis section provides a specification of the type mapping of C types used in Blender files to Java types, as used in Java .Blend. Scalar TypesThe following table contains mappings of all scalar types in Blender files (C) to Java.
Signed and Unsigned Integer TypesJava does not support unsigned integer types. Thus, every unsigned type is mapped to its signed counterpart. This has to be considered in case of overflows. 32 and 64 bit SystemsIn C, the size of pointers and the scalar types long and unsigned long depend on the system architecture: 8 byte on 64 bit systems and 4 byte on 32 bit systems. Both are mapped to 64 bit in Java .Blend. Structured TypesBased on the given type information from StructDNA, the model generator maps all structured types to Java facade classes. For each field of a struct a getter and setter method is generated based on type and name of the field. Embedded StructsIf a field of a struct is of a structured type (not a pointer) then it is embedded in the struct in C. Those fields get represented by a facade of the fields type. The facade returned by the corresponding getter method provides access to the embedded struct. When assigning another object (i.e. facade) to the field through its set method, the value of that object is copied into the memory region of the embedded struct. This is exactly the same behaviour as in C. PointersPointers in C can refer to different kinds of data which has to be known to understand the mapping chosen here. Generally, a pointer can be interpreted as a reference on a single data object or an array of data objects. /* pointer on single elem */ int i; int* pi = &i; // same output from both printf statments printf("%d", *pi); printf("%d", pi[0]); /* pointer on array of elems */ int a[] = {1,2,3}; int* pa = a; // same output from both printf statments printf("%d", pa[1]); printf("%d", *(pa+1)); This small example displays a couple of issues for type mapping of pointers to Java. Java builtin types can only deal with a few of the functionalities provided by a pointer, such as arrays of fixed length or references on objects. Often in C a pointer is used as reference on a set of elements in an array, which ends with one specific terminating element, such as a null value. Every string in C is stored that way. To access its data you have to iterate through the array and scan for the terminating element. The only case you can scan through data in Java using builtin types, is that you have an array. Unfortunately, to create an array, you first need its length, that means you can't solve it without native access to data. Another issue is, that scalar types such as int, float, etc., cannot be referenced. Additionally, the corresponding class types such as Another case are type casts applied to pointers. In C it is possible to cast memory to any type. Best example are This brief discussion of the different functionalities of a pointer in C shows that it is not possible to map a pointer type to a particular Java builtin type without semantic knowledge about its use case. But the developer using the pointer can easily determin its use case and just needs a suitable interface which supports all the functions of a C pointer in Java. In order to keep the data model generic Java .Blend introduces a Class
|
key | value | type | description | ||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
system | "Blender" | String | Always "Blender". | ||||||||||||||||||||||||||||
module | "DNA" | String | Always "DNA". | ||||||||||||||||||||||||||||
version | major.minor | String | Version string as retrieved from file header (e.g. "2.69") | ||||||||||||||||||||||||||||
source | source of information | String | String which specifies where this information was gathered from (e.g. Python API, Blender Source Code, etc.) | ||||||||||||||||||||||||||||
inherits | list of doc files to be inherited | Array of Strings | List of paths to doc files to be referred to, in case no other documentation was found for a given subject (e.g. struct). Inherited documentation will be overridden by included documentation or documentation found in the given doc file. | ||||||||||||||||||||||||||||
includes | list of files to be included | Array of Strings | List of paths to doc files to be included in this documentation file. This will override previously inherited or included documentation. | ||||||||||||||||||||||||||||
doc | Reserved for javadoc package info (e.g. package.html) | String | Place for package documentation which is not used in the current version. | ||||||||||||||||||||||||||||
structs | list of structs | Array of JSON objects | List of structs with documentation.
|
All elements which contain Javadoc comments (i.e. all properties with key 'doc'
) contain text only. That means, no slashes or stars like '/**' or '*/'. Since Javadoc accepts HTML complex constructs, like tables or lists can be added with HTML code.
dnadoc
Main folder of the DNA documentation.
<majorVer.minorVer>
Version subfolder (e.g. 2.69
).
dnasrc
Documentation retrieved from Blender source code via doxygen.
doc.json
Documentation entry file.pyapi
Documentation retrieved from Blender's Python API source code via script rna_info.py
.
doc.json
Documentation entry file.added
Manually added documentation.
doc.json
Documentation entry file.DNA_ID.json
Documentation for all structs in DNA_ID.h
DNA_object_types.json
Documentation for all structs in file DNA_object_types.h
Documentation provider is an interface providing lookup of documentation stored in files of the documentation system. The documentation provider automatically searches the most appropriate documentation for a given Blender file version combination (i.e. MINVERSION
and VERSION
and their respective extensions (subversions)).
Since there are multiple sources of documentation (i.e. Python API, Blender Source Code and manually added ("Java .Blend")) the documentation provider combines them in a list of documentation entries where each source gets its own section.
This is a brief getting started guide for Java .Blend API programmers. It covers just the basic tasks such as setting up the development environment, and reading or writing Blender files with help of the class BlenderFactory
.
To give a more detailed view on Java .Blend, all of its features have been demonstrated in examples which can be found in the download section on the Java .Blend website.
As explained in the concept overview, working with Java .Blend requires a set of facade classes and utilities which are generated by the code generator. You can either use a pregenerated Java archive of the download section or run the code generator on your own (see Section Generating Data Model and Utilities). The latter case is necessary if there is either no pregenerated Java archive available for your Blender version or you decided to work on code generator or documentation.
To get running, add the jar of the pregenerated data model to your project:
JavaBlend-<version>-DNA-<blender-version>.jar
There is an easy way and a more complex but even more flexible way to read Blender files. The easiest way is to use the class MainLib
and the more complex involves direct access to blocks.
MainLib
BlenderFile
. Class MainLib
provides access to all library elements found in the file and those provide access to the remaining raw data through its getter methods.
BlenderFile f = new BlenderFile(new File(filename)); MainLib lib = new MainLib(f); f.close(); // all data was read by MainLib constructor Mesh mesh = lib.getMesh(); // access mesh
Reading with direct block access basically involves opening a Blender file, retrieve/search blocks, instantiate appropriate facade classes based on the type information (sdnaIndex) given in the block and finally access it. See example class ExampleConvert2Json
for a comprehensive example on direct block access including meta data lookup etc.
Creating or extending Blender files requires a lot of knowledge on the Blender data model. Unfortunately, the data model is quite huge and contains lots of references between elements and redundant information too. The best way to gather information on how to write a Blender file, is to study one first and use it as template.
As part of the examples, you will find a class called ExampleConvert2Json
, which creates a JSON file with all information of a Blender file in the given block order. This file is quite easy to read and helps a lot in studying the Blender DNA.
Additionally, you will need to download the Blender sources and open the folder source/blender/makesdna
. There, you will find various C header files with the prefix DNA_
. Those header files contain all types (structs) of the Blender DNA (i.e. all types that can possibly exist in a Blender file). When not sure about the meaning of a member variable of a struct, then look if there is source code documentation and if you have an IDE which supports function call lookup etc. (i.e. Eclipse), then use it to find contexts in which this member variable is accessed.
The example CopybufferExchange
provides a full example on how to create a Blender file for exchange via the clipboard functionality of Blender.
As mentioned, the documentation on the generated facade classes is still poor and programmers will need to read Blenders source code and study .blend
files to gather more information. To speed things up, everybody is encouraged to contribute to the externally maintained documentation. This is not just about helping others, it is more about helping each other in terms of a team effort! If we all study the Blender source code for ourselves, we waste a tremendous amount of work power, trying to find knowledge over and over again, someone else already figured out! Just do the math.
The DNA documentation is also available on github. Its URL is:
https://github.com/homacs/org.cakelab.blender.dnadoc.git
To contribute to the documentation just fork it on github, and send me a pull request with your modifications. Contributions are reviewed and incorporate on a regular basis.
The format of the documentation is explained in Section Externally Maintained Documentation. Please check your modifications by performing a code generation run (see Section Generating Data Model and Utilities).
This section explains how to generate Java .Blend facade classes and utilities for a given Blender version. This procedure is 'almost' generic. In case you experience issues, then read the Section Troubleshooting below.
The generated code is subdivided into two packages:
The utilities package is slightly more version dependent than the facade classes. That means, the code generator may generate an incompatible utilities package in future versions of Blender while the facade classes will most certainly remain compatible and working. Therefore, the generation of the utilities package is optional.
In order to generate the facade classes you need a .blend
file saved with the version of Blender you are aiming to support as reference. This can be any .blend
file and even the user preferences to be found in $HOME/.config/blender/2.69/config/userpref.blend
.
In order to incorporate Javadoc comments in the generated facade classes you need to download a set of Java .Blend's external documentation files either directly from the github repository or the download section of Java .Blend's website. Please note, that you have to respect the license of the documentation files if you plan to redistribute derived work. Please refer to Section 'Licensing' on the index page to learn more about the reasons.
The code generator gets started from command line.
java -jar JavaBlend-SDK-<version>.jar <options>
Alternatively, you can use the Ant build file export-DNA-lib.xml
and modify it to adjust it to your environment.
name | type | default | description |
---|---|---|---|
-in | path | Blender file (always containing type information) to use as reference. | |
-out | path | Source code folder to create the package and store the generated classes. | |
-p | Java package name | "org.cakelab.blender" | Main package of generated code. Facades will go in subpackage 'dna' and utilities in 'utils'. |
-u | boolean | true | Enable (true) or disable (false) generation of utilities package. |
-c | path | "resources/dnadoc" | Main folder of the documentation system. This is optional. If no documentation is found, the code generator will still generate all code. |
-d | boolean | true | Enable (true) or disable (false) debug output. It's currently used by the documentation system only. |
-h | Prints brief help message and exits without code generation. |
> java -jar JavaBlend-SDK-1.0.0.jar -in "$HOME/.config/blender/2.69/config/userpref.blend" -out "../MyProject/src"
As you see, you can even use the user preferences file as reference .blend
file to provide the type information needed to generate the code.