I am using Jackson 1.9. in my web application wherein I require to convert complex objects e.g Spring’s ModelMap, BindingResult, java.uil.Map to JSON String objects.
Please consider the following code snippet where I am attempting one such conversion:
Map<String, Object> methodArgsMap = new HashMap<String, Object>();
methodArgsMap.put("map", map);/*map is an instance of org.springframework.ui.ModelMap*/
methodArgsMap.put("command", command);/*command is an instance of a custom POJO viz.ReportBeanParam*/
methodArgsMap.put("result", result);/*result is an instance of org.springframework.validation.BindingResult*/
The method is JSONProcessUtil. getObjectsAsJSONString(...) implemented as follows :
public final class JSONProcessUtil {
private static ObjectMapper objectMapper;
static {
objectMapper = new ObjectMapper();
/*Start : Configs. suggested by Jackson docs to avoid OutOfMemoryError*/
SerializationConfig serConfig = objectMapper.getSerializationConfig();
serConfig.disable(SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS);
objectMapper.getJsonFactory().configure(
JsonParser.Feature.INTERN_FIELD_NAMES, false);
objectMapper.getJsonFactory().configure(
JsonParser.Feature.CANONICALIZE_FIELD_NAMES, false);
/*End : Configs. suggested by Jackson docs to avoid OutOfMemoryError*/
}
public static Map<String, String> getObjectsAsJSONString(
Map<String, Object> argsMap) throws JsonGenerationException,
JsonMappingException, IOException {
log.info("Source app.In JSONProcessUtil.getObjectsAsJSONString(...)");
Map<String, String> jsonStrMap = null;
if (!(argsMap == null || argsMap.isEmpty())) {
jsonStrMap = new HashMap<String, String>();
Set<String> keySet = argsMap.keySet();
Iterator<String> iter = keySet.iterator();
String argName = null;
while (iter.hasNext()) {
argName = iter.next();
log.info("Source app. argName = {}, arg = {} ", argName,
argsMap.get(argName));
jsonStrMap.put(argName,
objectMapper.writeValueAsString(argsMap.get(argName)));/*The line giving error*/
log.info("Proceeding to the next arg !");
}
}
log.info("Source app. Exit from JSONProcessUtil.getObjectsAsJSONString(...)");
return jsonStrMap;
}
}
I am getting an OutOfMemoryError as follows :
INFO [http-8080-7] (JSONProcessUtil.java:73) - Source app. argName = result, arg = org.springframework.validation.BeanPropertyBindingResult: 0 errors DEBUG [http-8080-7] (SecurityContextPersistenceFilter.java:89) - SecurityContextHolder now cleared, as request processing completed Feb 20, 2012 5:03:30 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet saas threw exception
java.lang.OutOfMemoryError: Java heap space
at org.codehaus.jackson.util.TextBuffer._charArray(TextBuffer.java:
674)
at org.codehaus.jackson.util.TextBuffer.expand(TextBuffer.java:633)
at org.codehaus.jackson.util.TextBuffer.append(TextBuffer.java:438)
at org.codehaus.jackson.io.SegmentedStringWriter.write(SegmentedStringWriter.java:69)
at org.codehaus.jackson.impl.WriterBasedGenerator._flushBuffer(WriterBasedGenerator.java:1810)
at org.codehaus.jackson.impl.WriterBasedGenerator._writeFieldName(WriterBasedGenerator.java:345)
at org.codehaus.jackson.impl.WriterBasedGenerator.writeFieldName(WriterBasedGenerator.java:217)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:426)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142)
at org.codehaus.jackson.map.ser.impl.ObjectArraySerializer.serializeContents(ObjectArraySerializer.java:121)
at org.codehaus.jackson.map.ser.impl.ObjectArraySerializer.serializeContents(ObjectArraySerializer.java:28)
at org.codehaus.jackson.map.ser.ArraySerializers$AsArraySerializer.serialize(ArraySerializers.java:56)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:428)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:428)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142)
at org.codehaus.jackson.map.ser.MapSerializer.serializeFields(MapSerializer.java:287)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:212)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:23)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:428)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142)
at org.codehaus.jackson.map.ser.MapSerializer.serializeFields(MapSerializer.java:287)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:212)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:23)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:428)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142)
at org.codehaus.jackson.map.ser.MapSerializer.serializeFields(MapSerializer.java:287)
Please guide about resolving the same.
Thanks and regards !
Sounds like you are producing a huge JSON output, which gets buffered in memory.
This based on error message.
Your choices are either:
Use streaming output to avoid buffering it in memory (however, I am not sure if Spring allows you to do this), or
Increase heap size so you have enough memory
Features to disable interning and canonicalization are only relevant for parsing, and you are generating JSON, not parsing.
Related
I'm new to contract Testing Automation and I've written my first test using jvm-pact. I'm using junit5.
Below is the code
#ExtendWith(PactConsumerTestExt.class) #PactTestFor(providerName = "testProvider", port = "8081") public class ConsumerTests {
public static final String EXPECTED_BODY = "/integration/stubs/team_members/SingleTeamMember.json";
#Pact(consumer = "testConsumer" , provider="testProvider")
public RequestResponsePact singleTeamMemberSuccess(PactDslWithProvider builder) {
Map<String, String> headers = new HashMap<>();
headers.put("Content-Type", "application/json");
return builder
.given("I have at least one team member")
.uponReceiving("a request for a single team member")
.path("/team-members/1")
.method("GET")
.willRespondWith()
.status(200)
.headers(headers)
.body(EXPECTED_BODY)
.toPact();
}
#Test
#PactTestFor(pactMethod = "singleTeamMemberSuccess")
void testSingleTeamMemberSuccess(MockServer mockServer) throws IOException {
HttpResponse httpResponse = (HttpResponse) Request.Get(mockServer.getUrl() + "/team-members/1")
.execute().returnResponse();
assertThat(httpResponse.getStatusLine().getStatusCode(), is(equalTo(200)));
//assertThat(httpResponse.getEntity().getContent(), is(equalTo(TeamMemberSingle200.EXPECTED_BODY_SINGLE_TEAM_MEMBER)) );
}
I'm getting below error on running mvn install
ConsumerTests The following methods annotated with #Pact were not executed during the test: ConsumerTests.singleTeamMemberSuccess If these are currently a work in progress, and a #Disabled annotation to the method
[ERROR] ConsumerTests.singleTeamMemberSuccess:42 » NoClassDefFound Could not initialize class org.codehaus.groovy.reflection.ReflectionCache
Please can someone take a look and advise if I'm missing anything important to run the test successfully.
Thanks,
Poonam
I am using jdbcTemplate to query hive then writing the results to a .csv file. I basically just generate a list of objects then steam the list to write each record to the file.
I will like to stream the results as they coming back from hive and write it to the file instead of wait to get the whole thing then processing it. Can anyone pointing me to the right direction? Thanks!
private List<Avs> queryAvsData(String asSql) {
List<Avs> llistAvs = new ArrayList<Avs>();
List<Map<String, Object>> rows = hiveJdbcTemplate.queryForList(asSql);
Iterator<Map<String, Object>> it = rows.iterator();
while (it.hasNext()) {
Map<String, Object> row = it.next();
Avs laAvs = Avs.builder()
.make((String) row.get("make"))
.model((String) row.get("model"))
.build();
llistAvs.add(laAvs);
}
return llistAvs;
}
It doesn't look like there's a built-in solution, but you can do it. Basically, you wrap the existing functionality in an iterator, and use a spliterator to turn it into a stream. Here's a blog post on the subject:
The code implements Spring’s ResultSetExtractor interface, which is a Single Abstract Method (SAM) interface, allowing the use of a lambda expression to implement it.
The implementation wraps the SQL ResultSet in an iterator, constructs a stream using the Spliterators and StreamSupport utility classes, and applies that to a Function taking a stream of row sets and returning a generic result.
It's possible to stream values from JdbcTemplate. The following example is a service based on Spring Boot 2.4.8.
As, I run into problems (connection leak) using queryForStream then I will put a demo code here just to know that stream must be closed after usage.
import lombok.RequiredArgsConstructor;
import org.springframework.jdbc.core.SingleColumnRowMapper;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.stereotype.Service;
import java.util.Map;
import java.util.stream.Stream;
#Service
#RequiredArgsConstructor
public class DataCleaningService {
private final NamedParameterJdbcTemplate jdbcTemplate;
public void doSomeStreaming() {
String nativeQuery = "SELECT string_value FROM my_table WHERE column = :valueToFiler";
Map<String, Object> queryParameters = Map.of("valueToFiler", "my value");
SingleColumnRowMapper<String> stringRowMapper = SingleColumnRowMapper.newInstance(String.class);
try (Stream<String> stringValueStream = jdbcTemplate.queryForStream(nativeQuery, queryParameters, stringRowMapper)) {
stringValueStream.forEach(stringValue -> {
// do the needed action with the value
//..
System.out.printf("My cool value: %s", stringValue);
});
}
}
}
I am working on a usecase where I am supposed to poll S3 -> read the stream for the content -> do some processing and upload it to another bucket rather than writing the file in my server.
I know I can achieve it using S3StreamingMessageSource in Spring aws integration but the problem I am facing is that I do not know on how to process the message stream received by polling
public class S3PollerConfigurationUsingStreaming {
#Value("${amazonProperties.bucketName}")
private String bucketName;
#Value("${amazonProperties.newBucket}")
private String newBucket;
#Autowired
private AmazonClientService amazonClient;
#Bean
#InboundChannelAdapter(value = "s3Channel", poller = #Poller(fixedDelay = "100"))
public MessageSource<InputStream> s3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(bucketName);
messageSource.setFilter(new S3PersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
return messageSource;
}
#Bean
#Transformer(inputChannel = "s3Channel", outputChannel = "data")
public org.springframework.integration.transformer.Transformer transformer() {
return new StreamTransformer();
}
#Bean
public S3RemoteFileTemplate template() {
return new S3RemoteFileTemplate(new S3SessionFactory(amazonClient.getS3Client()));
}
#Bean
public PollableChannel s3Channel() {
return new QueueChannel();
}
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.handle(streamFile())
.get();
}
}
Can someone please help me with the code to process the stream ?
Not sure what is your problem, but I see that you have a mix of concerns. If you use messaging annotations (see #InboundChannelAdapter in your config), what is the point to use the same s3InboundStreamingMessageSource in the IntegrationFlow definition?
Anyway it looks like you have already explored for yourself a StreamTransformer. This one has a charset property to convert your InputStreamfrom the remote S3 resource to the String. Otherwise it returns a byte[]. Everything else is up to you what and how to do with this converted content.
Also I don't see reason to have an s3Channel as a QueueChannel, since the start of your flow is pollable anyway by the #InboundChannelAdapter.
From big height I would say we have more questions to you, than vise versa...
UPDATE
Not clear what is your idea for InputStream processing, but that is really a fact that after S3StreamingMessageSource you are going to have exactly InputStream as a payload in the next handler.
Also not sure what is your streamFile(), but it must really expect InputStream as an input from the payload of the request message.
You also can use the mentioned StreamTransformer over there:
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.transform(Transformers.fromStream("UTF-8"))
.get();
}
And the next .handle() will be ready for String as a payload.
Given this entry in application.properties:
server.port=0
which causes Spring Boot to chose a random available port, and testing a spring boot web application using spock, how can the spock code know which port to hit?
Normal injection like this:
#Value("${local.server.port}")
int port;
doesn't work with spock.
You can find the port using this code:
int port = context.embeddedServletContainer.port
Which for those interested in the java equivalent is:
int port = ((TomcatEmbeddedServletContainer)((AnnotationConfigEmbeddedWebApplicationContext)context).getEmbeddedServletContainer()).getPort();
Here's an abstract class that you can extends which wraps up this initialization of the spring boot application and determines the port:
abstract class SpringBootSpecification extends Specification {
#Shared
#AutoCleanup
ConfigurableApplicationContext context
int port = context.embeddedServletContainer.port
void launch(Class clazz) {
Future future = Executors.newSingleThreadExecutor().submit(
new Callable() {
#Override
public ConfigurableApplicationContext call() throws Exception {
return (ConfigurableApplicationContext) SpringApplication.run(clazz)
}
})
context = future.get(20, TimeUnit.SECONDS);
}
}
Which you can use like this:
class MySpecification extends SpringBootSpecification {
void setupSpec() {
launch(MyLauncher.class)
}
String getBody(someParam) {
ResponseEntity entity = new RestTemplate().getForEntity("http://localhost:${port}/somePath/${someParam}", String.class)
return entity.body;
}
}
The injection will work with Spock, as long as you've configured your spec class correctly and have spock-spring on the classpath. There's a limitation in Spock Spring which means it won't bootstrap your Boot application if you use #SpringApplicationConfiguration. You need to use #ContextConfiguration and configure it manually instead. See this answer for the details.
The second part of the problem is that you can't use a GString for the #Value. You could escape the $, but it's easier to use single quotes:
#Value('${local.server.port}')
private int port;
Putting this together, you get a spec that looks something like this:
#ContextConfiguration(loader = SpringApplicationContextLoader, classes = SampleSpockTestingApplication.class)
#WebAppConfiguration
#IntegrationTest("server.port=0")
class SampleSpockTestingApplicationSpec extends Specification {
#Value("\${local.server.port}")
private int port;
def "The index page has the expected body"() {
when: "the index page is accessed"
def response = new TestRestTemplate().getForEntity(
"http://localhost:$port", String.class);
then: "the response is OK and the body is welcome"
response.statusCode == HttpStatus.OK
response.body == 'welcome'
}
}
Also note the use of #IntegrationTest("server.port=0") to request a random port be used. It's a nice alternative to configuring it in application.properties.
You could do this too:
#Autowired
private org.springframework.core.env.Environment springEnv;
...
springEnv.getProperty("server.port");
I am generating a PDF file via fop 1.0 out of a java library. The unit tests are running fine and the PDF is rendered as expected, including an external graphic:
<fo:external-graphic content-width="20mm" src="url('images/image.png')" />
If I render this within a Java EE application in glassfish 3.1, I always get the following error:
Image not found. URI: images/image.png. (No context info available)
I double-checked whether the image is available. It is available within the .jar file in the .ear file and should therfore be available by the ClasspathUriResolver. This is a code-snipplet of how I setup the fop-factory:
FopFactory fopFactory = FopFactory.newInstance();
URIResolver uriResolver = new ClasspathUriResolver();
fopFactory.setURIResolver(uriResolver);
Fop fop = fopFactory.newFop(MimeConstants.MIME_PDF, out);
...
I also assigned the URI resolver to the TransformerFactory and the Transformer with no success. Would be great if someone can help me out.
-- Wintermute
Btw: the ClasspathUriResolver() looks like this
public class ClasspathUriResolver implements URIResolver {
#Override
public Source resolve(String href, String base) throws TransformerException {
Source source = null;
InputStream inputStream = ClassLoader.getSystemResourceAsStream(href);
if (inputStream != null) {
source = new StreamSource(inputStream);
}
return source;
}
}
You consider a different class loader then ClassLoader.getSystemResourceAsStream(href);
Try InputStream inputStream = getClass().getResourceAsStream(href); or something else, maybe.
Does it work, then?