cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

jdhao
by New Contributor II
  • 2133 Views
  • 4 replies
  • 0 kudos

Why can't I query a table from a cluster, but can query from another cluster in the same workspace

I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`.  It shows some permission errors. Under cluster B, if I run the same sql query, it ...

  • 2133 Views
  • 4 replies
  • 0 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 0 kudos

Check for any spark config or init script differences in the two clusters.

  • 0 kudos
3 More Replies
gopikrsna925
by New Contributor
  • 1325 Views
  • 0 replies
  • 0 kudos

Azure Databricks: leading zeros in decimal integer literals are not permitted

Hey team,Need your help.I am trying to run the below python code in a data bricks notebook, which is part of parsing an XML file, exploding the element. This works great for all the other elements with no numbers and elements not starting with a zero...

Data Engineering
leading
zeros
  • 1325 Views
  • 0 replies
  • 0 kudos
Phani1
by Valued Contributor
  • 646 Views
  • 1 replies
  • 0 kudos

Streaming tables vs DLT

Have a couple of questions wrt Streaming tables, Kindly help us with this.1)Can we create streaming tables without a DLT pipeline?2)Can we create streaming tables in Databricks SQL?3)What we observe in Streaming tables, it  support Kafka and event lo...

Data Engineering
dlt
sql
Streaming tables
  • 646 Views
  • 1 replies
  • 0 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 0 kudos

HiPlease refer to the document: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html#create-streaming-tableI think this should help you answer your questions.

  • 0 kudos
thomasthomas
by New Contributor II
  • 5869 Views
  • 4 replies
  • 0 kudos

How can I insert into 2 tables within one database transaction with spark SQL / pyspark?

Hi all,I have a postgres database that contains two tables: A and B.Also, I have 2 delta tables, called C and D. My task is to push the data from A to C and B to D - and if something fails, then leave everything as is.With python it is easy. Set up t...

Data Engineering
Spark JDBC Commit
  • 5869 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @thomasthomas  We haven't heard from you since the last response from @daniel_sahal ​, and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to oth...

  • 0 kudos
3 More Replies
handreassa
by New Contributor III
  • 2065 Views
  • 3 replies
  • 1 kudos

Resolved! How to manage cluster permissions

How to manage different workspaces’ clusters permissions?

  • 2065 Views
  • 3 replies
  • 1 kudos
Latest Reply
KoenZandvliet
New Contributor III
  • 1 kudos

Which {request_object_type} to use for setting permissions for a cluster? "cluster", "clusters", "compute" does not work.

  • 1 kudos
2 More Replies
malati
by New Contributor II
  • 974 Views
  • 2 replies
  • 0 kudos

Unable to login or reset password Community Edition

I'm not able to login databricks community edition from last 5 days, getting  Invalid email address or password issue even though its correct credentials. i tried to reset password but unable to reset password also. what is the issue. can you guys pl...

  • 974 Views
  • 2 replies
  • 0 kudos
Latest Reply
malati
New Contributor II
  • 0 kudos

im getting mails saying that if issue is fixed then mark one as a accepted solution but unfortunately issue is still not yet fixed , can someone look into it and fix issue please ??  

  • 0 kudos
1 More Replies
bstanie
by New Contributor
  • 953 Views
  • 1 replies
  • 0 kudos

A problem with formating python code to pep8

Hi there,Anyone here experienced any problems with formatiing python code inside databricks notebook cells? I'm constantly having this weird problem, both with very short/simple and complex code, in different notebooks. Sometimes it somehow works, an...

  • 953 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

that is interesting.Can you check if you you use a version > 11.2.  If not you have to install black and tokenize-rt.Can you check if you use a custom pyproject.toml file?  It might be related to that.(You also need edit permission on the notebook bu...

  • 0 kudos
dzm
by New Contributor
  • 1010 Views
  • 1 replies
  • 0 kudos

Using Libreoffice in Databricks

Hi Community, I'm using Databricks E2, and need to convert pptx files to pdf files.This can be done in either a python or an R notebook using #LibreofficeTo achieve this I'd have to download LibreOffice; I'm not too sure on how to do that. Would I ha...

Data Engineering
pdf
pptx
python
R
  • 1010 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I suppose by Libreoffice you mean the sdk, without the frontend?You will have to install the jar as a library on the compute cluster.From that moment on, you can use the classes in your code.If you cannot run the jar from a command line, it might be ...

  • 0 kudos
emanuelsh
by New Contributor
  • 725 Views
  • 0 replies
  • 0 kudos

Schema Evolution from Kafka Source

Hi,I have a Spark streaming process that reads data from a Kafka topic to Azure DLThis is how I implement the MERGE capability into the delta table.In addition to the same topic, I have another streaming process that simply writes data to DLIn kafka ...

  • 725 Views
  • 0 replies
  • 0 kudos
katedb
by New Contributor
  • 1001 Views
  • 1 replies
  • 0 kudos

Clusters do not start - bootstrap timeout

Hello,Whenever I try to start any of already existing clusters, I get Bootstrap timeout error. In the logs, there are following messages:[Bootstrap Event] Can reach databricks-update-oregon.s3.us-west-2.amazonaws.com: [FAILED] [ 257.556698] audit: k...

Data Engineering
bootstrap
compute
  • 1001 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16752239289
Valued Contributor
  • 0 kudos

The error message indicate the ec2 instance cannot access databricks-update-oregon.s3.us-west-2.amazonaws.com. Do you have s3 endpoint setup or can traffic route to databricks-update-oregon.s3.us-west-2.amazonaws.com ?

  • 0 kudos
NWIEFInance
by New Contributor
  • 1184 Views
  • 1 replies
  • 0 kudos

Connect to EXCEL

I have hardtime connecting my existing EXCEL file to source data from DataBricks and need help

  • 1184 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16539034020
Contributor II
  • 0 kudos

Hi, Thanks for contacting Databricks Support. We doesn't support direct Excel-Databricks connectivity. However, Databricks can be accessed through ODBC and JDBC interfaces, and we can leverage these with Excel's Power Query functionality for indirect...

  • 0 kudos
matanper
by New Contributor III
  • 2847 Views
  • 5 replies
  • 1 kudos

Custom docker image fails to initalize

I'm trying to use a custom docker image for my job. This is my docker file:FROM databricksruntime/standard:12.2-LTS COPY . . RUN /databricks/python3/bin/pip install -U pip RUN /databricks/python3/bin/pip install -r requirements.txt USER rootMy job ...

  • 2847 Views
  • 5 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, I think, disabling iptables will be better in this case, could you please try the below command and confirm? $ sudo iptables -S

  • 1 kudos
4 More Replies
Łukasz
by New Contributor III
  • 3174 Views
  • 6 replies
  • 5 kudos

Resolved! Dense rank possible bug

I have the case of deduplicating data source over specific business key using dense_rank function. Currently the data source does not have any duplicates, so the function should return 1 in all cases. The issue is that dense rank does not return prop...

  • 3174 Views
  • 6 replies
  • 5 kudos
Latest Reply
saipujari_spark
Valued Contributor
  • 5 kudos

Hey @Łukasz Thanks for reporting.As I see Spark 3.4.0 introduced an improvement that looks to be the cause for this issue.Improvement: https://issues.apache.org/jira/browse/SPARK-37099Similar Bug: https://issues.apache.org/jira/browse/SPARK-44448This...

  • 5 kudos
5 More Replies
415963
by New Contributor II
  • 2142 Views
  • 3 replies
  • 2 kudos

Not able to catch structured streaming exception

I would like to catch and handle an exception in a structured streaming job.The databricks notebook still displays the exception, regardless of added exception handling (see attached screenshot)I guess that the exception is displayed by the cell outp...

  • 2142 Views
  • 3 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, I understand, could you please also provide the last line of the error after scrolling down in the notebook cell? 

  • 2 kudos
2 More Replies
Retko
by Contributor
  • 5445 Views
  • 4 replies
  • 2 kudos

Running Command is often stuck on "Running Command..."

Hi,when running command, it often gets stuck and message below it says: "Running Command..."What can I do with it besides of restarting cluster?Also tried reattaching and clearing state, but no help here.Thanks

  • 5445 Views
  • 4 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, do you see this while running a command in the notebook? Please tag @Debayan  with your next comment which will notify me. Thanks!

  • 2 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels