cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
tgen
by • New Contributor II
  • 1304 Views
  • 1 replies
  • 0 kudos

Increase stack size Databricks

Hi everyoneI'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this...

Data Engineering
Reply
  • 1304 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi everyoneI'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
satishnavik
by • New Contributor II
  • 8175 Views
  • 5 replies
  • 0 kudos

How to connect Databricks Database with Springboot application using JPA

facing issue with integrating our Spring boot JPA supported application with Databricks.Below are the steps and setting we did for the integration.When we are starting the spring boot application we are getting a warning as :HikariPool-1 - Driver doe...

Data Engineering
Reply
  • 8175 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
172036
New Contributor II
  • 0 kudos

Was there any resolution to this?  Is Spring datasource supported now?

  • 0 kudos
4 More Replies
JameDavi_51481
by • New Contributor III
  • 5257 Views
  • 8 replies
  • 0 kudos

Can we add tags to Unity Catalog through Terraform?

We use Terraform to manage most of our infrastructure, and I would like to extend this to Unity Catalog. However, we are extensive users of tagging to categorize our datasets, and the only programmatic method I can find for adding tags is to use SQL ...

Data Engineering
Reply
  • 5257 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
dbruehlmeier
Contributor
  • 0 kudos

Having tags with terraform would help a lot. Add them on cluster and schema level is crucial. Looking forward for an update about the open PR.

  • 0 kudos
7 More Replies
djburnham
by • New Contributor III
  • 1701 Views
  • 2 replies
  • 1 kudos

Resolved! How to get a list of workspace users who have the "unrestricted cluster create" entitlement ?

Hello - I'm hoping somebody can help me with this ... I have a lot of users configured with access to a workspace (100's) and I want to write a report to see if any of the users have  "unrestricted cluster create" entitlement in the workspace. This i...

Data Engineering
Reply
  • 1701 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
djburnham
New Contributor III
  • 1 kudos

Many thanks for you help @Yeshwanth  it put me on the right track. The API does have a filter option and that looks like it complies with rfc7644 but my attempts to use it were rather hit and miss - I suspect as the API is preview it is not fully imp...

  • 1 kudos
1 More Replies
Anonymous
by • Not applicable
  • 6541 Views
  • 11 replies
  • 2 kudos

Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection

HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. ​Is it by design? Is it a limitation when using Privat...

Data Engineering
Reply
  • 6541 Views
  • 11 replies
  • 2 kudos
View Replies
Latest Reply
RomanLegion
New Contributor III
  • 2 kudos

Fixed, go to Profile -> Compute->  SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.

  • 2 kudos
10 More Replies
jenshumrich
by • Contributor
  • 2158 Views
  • 1 replies
  • 0 kudos

Resolved! R install - cannot open URL

Neither standard nor non standard repo seem available. Any idea how to debug/fix this? %r install.packages("gghighlight", lib="/databricks/spark/R/lib", repos = "http://cran.us.r-project.org") Warning: unable to access index for repository http://cra...

Data Engineering
Reply
  • 2158 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jenshumrich
Contributor
  • 0 kudos

%sh nc -zv cran.us.r-project.org 80 It was a network issue. These lines above proved it and the network administrators had to open the IPs.

  • 0 kudos
BobBubble2000
by • New Contributor II
  • 3471 Views
  • 4 replies
  • 0 kudos

Delta Live Tables with Common Data Model as source

Hi,I'm investigating whether it's possible to use Common Data Model CDM (in particular the Dynamics 365 exported csv and cdm files) as a Delta Live Tables data source? Can someone point me in the right direction?Thanks!

Data Engineering
Reply
  • 3471 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Suryanarayan
New Contributor II
  • 0 kudos

Using Delta Live Tables with Common Data Model (CDM) as a Source in DatabricksI'm investigating the use of Delta Live Tables (DLT) to process Common Data Model (CDM) files exported from Dynamics 365, and I found a solution that works well. Here’s a q...

  • 0 kudos
3 More Replies
Jackson1111
by • New Contributor III
  • 610 Views
  • 3 replies
  • 1 kudos

get job detail API

Hello, is there an API interface for passing in batches of run_id to obtain job running details?

Data Engineering
Reply
  • 610 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
mhiltner
Databricks Employee
  • 1 kudos

Maybe this could help. Its not batch, but you can get the run_id details  https://docs.databricks.com/en/workflows/jobs/jobs-2.0-api.html#runs-get-output

  • 1 kudos
2 More Replies
eva_mcmf
by • New Contributor II
  • 905 Views
  • 1 replies
  • 0 kudos

Autoloader with SQLite db files

Hi Everyone, Is it possible to ingest SQLite db files with Databricks Autoloader? Is it currently supported? If so, could you please share an example?

Data Engineering
autoloader
azure
ingestion
sqlite
Reply
  • 905 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
lucasrocha
Databricks Employee
  • 0 kudos

Hello @eva_mcmf , I hope this message finds you well. As per the documentation, Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3, Azure Data Lake Storage G...

  • 0 kudos
Chengcheng
by • New Contributor III
  • 2437 Views
  • 1 replies
  • 0 kudos

The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)

Hi everone, I'm trying to query data in Azure Synapse Dedicated SQL Pool according to the documentaion using:.format("com.databricks.spark.sqldw") Query data in Azure Synapse AnalyticsIt says that a abfss temporary location is needed.But I found that...

Data Engineering
Azure Synapse Connector
Data Ingstion
JDBC
Reply
  • 2437 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
lucasrocha
Databricks Employee
  • 0 kudos

Hello @Chengcheng , I hope this message finds you well. As per the documentation the "tempDir" parameter is a required one and there is no default value for it.Databricks Synapse connector options reference: https://docs.databricks.com/en/connect/ext...

  • 0 kudos
VJ3
by • New Contributor III
  • 1009 Views
  • 2 replies
  • 0 kudos

Databricks Upload local files (Create/Modify table)

Hello Team,I believe Databricks come out recently feature of Create or modify a table using file upload which is less than 2 GB (file format CSV, TSV, or JSON, Avro, Parquet, or text files to create or overwrite a managed Delta Lake table) on Self Se...

Data Engineering
Reply
  • 1009 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
VJ3
New Contributor III
  • 0 kudos

Hello Nandini,Thank you for reply. Apologies for delay. Let's say I uploaded CSV file containing PII data using Upload feature available in Databricks UI. Will I be able to share that file with another user who should not have access to PII data elem...

  • 0 kudos
1 More Replies
PabloCSD
by • Contributor II
  • 1438 Views
  • 4 replies
  • 1 kudos

Resolved! My Libraries are not being installed in dbx-pipelines

Hello,I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel f...

PabloFelipe_0-1717599844578.png
Data Engineering
Databricks
dbx
Reply
  • 1438 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
PabloCSD
Contributor II
  • 1 kudos

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

  • 1 kudos
3 More Replies
AH
by • New Contributor III
  • 459 Views
  • 1 replies
  • 0 kudos

Resolved! Delta Lake Table Daily Read and Write job optimization

I have created 7 job for each business system to extract product data from each postgress source then write all job data into one data lake delta table [raw_product].each business system product table has around 20 GB of data.do the same thing for 15...

AH_0-1717569489175.png AH_1-1717572455868.png AH_3-1717572644640.png AH_2-1717572557758.png
Data Engineering
Reply
  • 459 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 0 kudos

@AH  - we can try out the config  if read or fetch from postgres is slow , we can increase the fetchsize , numPartitions (to increase parallelism). kindly try to do a df.count() to check on slowness.  https://spark.apache.org/docs/latest/sql-data-sou...

  • 0 kudos
SamGreene
by • Contributor
  • 3849 Views
  • 4 replies
  • 0 kudos

Resolved! Using parameters in a SQL Notebook and COPY INTO statement

Hi, My scenario is I have an export of a table being dropped in ADLS every day.  I would like to load this data into a UC table and then repeat the process every day, replacing the data.  This seems to rule out DLT as it is meant for incremental proc...

Data Engineering
Reply
  • 3849 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
SamGreene
Contributor
  • 0 kudos

The solution that worked what adding this python cell to the notebook: %pythonfrom pyspark.dbutils import DBUtilsdbutils = DBUtils(spark)dbutils.widgets.text("catalog", "my_business_app")dbutils.widgets.text("schema", "dev") Then in the SQL Cell: CRE...

  • 0 kudos
3 More Replies
galzamo
by • New Contributor
  • 627 Views
  • 1 replies
  • 0 kudos

Job running time too long

Hi all,I'm doing my first data jobs.I create one job that consists of 4 other jobs.Yesterday I ran the 4 jobs separately and it worked fine (about half hour)-today I ran the big job, and the 4 jobs is running for 2 hours (and still running), Why is t...

Data Engineering
Reply
  • 627 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
anardinelli
Databricks Employee
  • 0 kudos

Hello @galzamo how are you? You can check on the SparkUI for long running stages that might give you a clue where it's spending the most time on each task. Somethings can be the reason: 1. Increase of data and partitions on your source data 2. Cluste...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels