- 11139 Views
- 1 replies
- 3 kudos
Resolved! Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes).
I got the below error when running a streaming workload from a source Delta table Caused by: java.lang.RuntimeException: Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes). As a workaround, you can reduce ...
- 11139 Views
- 1 replies
- 3 kudos
- 3 kudos
This is happening because the delta/parquet source has one or more of the following:a huge number of columnshuge strings in one or more columnshuge arrays/map, possibly nested in each otherIn order to mitigate this issue, could you please reduce spar...
- 3 kudos
- 7352 Views
- 1 replies
- 1 kudos
Resolved! Query Pushdown in Snowflake
Hi,I am wondering what documentation exists on Query Pushdown in Snowflake.I noticed that a single function (monitonically_increasing_id()) prevented the entire query being pushed down to Snowflake during an ETL process. Is Pushdown coming from the S...
- 7352 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Sam,The Spark Connector applies predicate and query pushdown by capturing and analyzing the Spark logical plans for SQL operations. When the data source is Snowflake, the operations are translated into a SQL query and then executed in Snowflake to...
- 1 kudos
- 9178 Views
- 5 replies
- 3 kudos
Resolved! Use Spot Instances with Azure Data Factory Linked Service
In my pipeline I'm using Azure Data Factory to trigger Databricks notebooks as a linked serviceI want to use spot instances for my job clusters Is there a way to achieve this?I didn't find a way to do this in the GUI.Thanks for your help!Marco
- 9178 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Werner Stinckens​ ,Just a quick follow up question.Does it make sense to you that you can select the following options in Azure Data Factory?To my understanding, "cluster version", "Python Version" and the "Worker options" are defined when I crea...
- 3 kudos
- 21405 Views
- 2 replies
- 3 kudos
spark-xml not working with Databricks Connect and Pyspark
Hi all,I currently have a cluster configured in databricks with spark-xml (version com.databricks:spark-xml_2.12:0.13.0) which was installed using Maven. The spark-xml library itself works fine with Pyspark when I am using it in a notebook within th...
- 21405 Views
- 2 replies
- 3 kudos
- 3 kudos
@Sean Owen​ I do not believe I have. Do you have any documentation on how to install spark-xml locally? I have tried the following with no luck. IS this what you are referring to?PYSPARK_HOME/bin/pyspark --packages com.databricks:spark-xml_2.12:0.13....
- 3 kudos
- 9768 Views
- 8 replies
- 14 kudos
Resolved! Real-time model serving and monitoring on Databricks at scale
How to deploy real-time model on databricks at scale? Right now, The model serving is very limited to 20 requests per second. Also, There are no model monitoring framework/graphs like the one's provided with AzureML or Sagemaker frameworks.
- 9768 Views
- 8 replies
- 14 kudos
- 14 kudos
I believe the next update to serving will include 1, not 2 (this is still within a Databricks workspace in a region). I don't think multi-model endpoints are on the roadmap next.How does Airflow integration relate?
- 14 kudos
- 3345 Views
- 1 replies
- 4 kudos
Embed Google Slides (PowerPoint) into Databricks Interactive Notebooks Use the following code to embed your slides:slide_id = '1CYEVsDqsdfg343fwg4...
Embed Google Slides (PowerPoint) into Databricks Interactive NotebooksUse the following code to embed your slides:slide_id = '1CYEVsDqsdfg343fwg42MtXqGd68gffP-Y16CR59c' slide_number = 'id.p9' displayHTML(f''' <iframe src="https://docs.google.com/...
- 3345 Views
- 1 replies
- 4 kudos
- 4 kudos
@Artem Yevtushenko​ - Thank you for sharing this solution.
- 4 kudos
- 16238 Views
- 5 replies
- 5 kudos
- 16238 Views
- 5 replies
- 5 kudos
- 4219 Views
- 1 replies
- 3 kudos
Resolved! Do Spark nodes read data from storage in a sequence?
I'm new to Spark and trying to understand how some of its components work.I understand that once the data is loaded into the memory of separate nodes, they process partitions in parallel, within their own memory (RAM).But I'm wondering whether the in...
- 4219 Views
- 1 replies
- 3 kudos
- 3 kudos
@Narek Margaryan​ , Normally the reading is done in parallel because the underlying file system is already distributed (if you use HDFS-based storage or something like, a data lake f.e.).The number of partitions in the file itself also matters.This l...
- 3 kudos
- 11370 Views
- 2 replies
- 3 kudos
Resolved! Query with distinct sort and alias produces error column not found
I’m trying to use sql query on azure-databricks with distinct sort and aliasesSELECT DISTINCT album.ArtistId AS my_alias FROM album ORDER BY album.ArtistIdThe problem is that if I add an alias then I can not use not aliased name in the order by ...
- 11370 Views
- 2 replies
- 3 kudos
- 4619 Views
- 1 replies
- 1 kudos
Resolved! upgraded R package rlang to 0.4.11 on DBR 8.3 SC, but sessionInfo() still shows rlang as 0.4.9
I am using Azure Databricks Runtime (DBR) 8.3 ML with Python notebook and R cells together.I want to use "tidyverse" and one of the dependency is rlang >= 0.4.10 and the base DBR 8.3 ML provides rlang @ 0.4.9. I successfully upgraded the R package t...
- 4619 Views
- 1 replies
- 1 kudos
- 1 kudos
@Jim Huang​ : Could you please try to restart the session and try to run tidyverse. Looks like the older version of rlang loaded in session.Error : package or namespace load failed for ‘tidyverse’ in loadNamespace(i, c(lib.loc, .libPaths()), versionC...
- 1 kudos
- 5226 Views
- 2 replies
- 5 kudos
Resolved! Is there a stable, ideally official JMS/ActiveMQ connector for Spark?
We're delivering pipelines that are mostly based on Databricks Spark Streaming, Delta Lake and Azure Event Hubs, and there's a requirement to integrate with AMQ/JMS endpoints (Request and Response queues in ActiveMQ).Is there a proven way to integrat...
- 5226 Views
- 2 replies
- 5 kudos
- 5 kudos
@amichel We have a feature request to add structured streaming support for Tibco EMS and JMS. Unfortunately, it's yet to be prioritized for the roadmap. I would request to file a feature request in our ideas portal https://ideas.databricks.com/ideas/...
- 5 kudos
- 14774 Views
- 7 replies
- 3 kudos
Resolved! Is there a way to automate Table creation in Databricks SQL based on a ADLS storage location which contains multiple Parquet files?
We have ADLS container location which contains several (100+) different data subjects folders which contain Parquet files with partition column and we want to expose each of the data subject folder as a table in Databricks SQL. Is there any way to au...
- 14774 Views
- 7 replies
- 3 kudos
- 3 kudos
Updating dazfuller suggestion, but including code for one level of partitioning, of course if you have deeper partitions then you will have to make a function and do a recursive call to get to the final directory containing parquet files. Parquet wil...
- 3 kudos
- 6747 Views
- 4 replies
- 2 kudos
Resolved! Slow first time run, jar based jobs
When running a jar-based job, I've noticed that the 1st run always takes the extra time to complete the job and consecutive runs take less time to finish the job. This behavior is reproducible on an interactive cluster. What's causing this? Is this e...
- 6747 Views
- 4 replies
- 2 kudos
- 2 kudos
@Sandeep Katta​ , this is a fat jar that does read-transform-write. @DD Sharma​ response matches @Werner Stinckens​ & I intuition that there was efficiency on the second job due to jar already being loaded. I would not have noticed this had job run...
- 2 kudos
- 4384 Views
- 4 replies
- 7 kudos
Resolved! Visualization of Structured Streaming in job.
Does Databricks have feature or good pattern to visualize the data from Structured Streaming? Something like display in the notebook.
- 4384 Views
- 4 replies
- 7 kudos
- 4033 Views
- 2 replies
- 6 kudos
Resolved! New Bill Inmon Book, What are your thoughts?
Have you checked out the new Bill Inmon Book, Building the Data Lakehouse? https://dbricks.co/3uxCXjO What were your thoughts if you read it?
- 4033 Views
- 2 replies
- 6 kudos
- 6 kudos
The quality of the book depends on the audience IMO. For people who have no background in data warehousing it will be interesting to read. For the others the book is too general and descriptive. The 'HOW do you do x' is missing.
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
