cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SailajaB
by • Valued Contributor III
  • 3263 Views
  • 2 replies
  • 5 kudos

An error occurred while calling o303.mount: Operation failed: "This request is not authorized to perform this operation

Hi Team,We are unable to mount storage container in below scenario We created Gen 2 using VNet and added firewall restrictions (i.e allow trusted sources)And deployed Data bricks workspace with out VNet injection. Is it possible to add databricks pub...

Data Engineering
Reply
  • 3263 Views
  • 2 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hey @Sailaja B​ Hope everything is great!Does Hubert's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thanks!

  • 5 kudos
1 More Replies
sheree
by • New Contributor III
  • 2917 Views
  • 3 replies
  • 1 kudos

Resolved! I can't access to my account.

I can't access to my account.This acccount was created today(not community, after 14 days trial it will chargable)when I'm try to access my account it gives meInvalid email address or passwordNote: Emails/usernames are case-sensitiveI tried to reset ...

Data Engineering
Reply
  • 2917 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sheree
New Contributor III
  • 1 kudos

I got a reset link from the community. Actually the problem was with my username ,it did not identify a character within my username which was my email id.

  • 1 kudos
2 More Replies
chandan_a_v
by • Valued Contributor
  • 16404 Views
  • 6 replies
  • 6 kudos

Resolved! Spark Driver Out of Memory Issue

Hi, I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue. Spark config :from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Demand Forecasting...

Data Engineering
Reply
  • 16404 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
chandan_a_v
Valued Contributor
  • 6 kudos

I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.

  • 6 kudos
5 More Replies
William_Scardua
by • Valued Contributor
  • 2639 Views
  • 1 replies
  • 2 kudos

Resolved! Best way to encrypt PII data

Hi guys, I have around 600GB per load, in you opnion, what is the best way to encrypt PII data in terms of performance ? (lib, cluster type, etc.)Thank youWilliam

Data Engineering
Reply
  • 2639 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 2 kudos

Hello @William Scardua​ please check if the blog helps you.https://databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html

  • 2 kudos
rahul3
by • New Contributor
  • 3032 Views
  • 1 replies
  • 1 kudos

Facing mount/unmount issue while running same job parallelly with scala.

 Using above configuration in cluster, when I run databricks job parallelly with multiple request at a same time, then I am getting mount/unmount issue. For an example : When I make three request to databricks job , it run 3 jobs parallelly but somet...

image.png image.png
Data Engineering
Reply
  • 3032 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

hi @rahul upadhyay​ are you using the same mount path /mnt/rahul in all the 3 jobs? Could you please add the full error message?

  • 1 kudos
Devarsh
by • Contributor
  • 9502 Views
  • 3 replies
  • 7 kudos

Resolved! Getting the error 'No such file or directory', when trying to access the json file

I am trying to write in my google sheet through Databricks but when it comes to reading the json, file containing the credentials, I am getting the error that No such file or directory exists.import gspread     gc = gspread.service_account(filename='...

Data Engineering
Reply
  • 9502 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 7 kudos

Hi @Devarsh Shah​ The issue is not with json file but the location you are specifying while reading.As suggested by @Werner Stinckens​ please start using spark API to read the json file as below:spark.read.format("json").load("testjson")Please check ...

  • 7 kudos
2 More Replies
BhagS
by • New Contributor II
  • 5746 Views
  • 2 replies
  • 5 kudos

Resolved! Write Empty Delta file in Datalake

hi all,Currently, i am trying to write an empty delta file in data lake, to do this i am doing the following:Reading parquet file from my landing zone ( this file consists only of the schema of SQL tables)df=spark.read.format('parquet').load(landingZ...

image
Data Engineering
Reply
  • 5746 Views
  • 2 replies
  • 5 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 5 kudos

Hi @bhagya s​ Since your source file is empty, there is no data file inside the centralizedZonePath directory i.e .parquet file is not created in the target location. However, _delta_log is the transaction log that holds the metadata of the delta for...

  • 5 kudos
1 More Replies
Krishscientist
by • New Contributor III
  • 2298 Views
  • 2 replies
  • 0 kudos

How to merge delta data..

Data from Parquet to delta converted and delta files written into diff folders based on SRC_SYS_ID....Any one help me how to merge delta data from multiple folders.Regards.

Data Engineering
Reply
  • 2298 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

Hi @Krishna Kommineni​ Is the table partitioned on SRC_SYS_ID col?

  • 0 kudos
1 More Replies
scholar
by • New Contributor II
  • 3101 Views
  • 3 replies
  • 2 kudos

How to read data from kafka topic using spark streaming

I have installed kafka-2.10-0.10.2. And using cluster with configuration: Runtime :6.4 Extended Support( scala 2.11,Spark 2.4.5) After this i am able to get mesgage son producer and consumer But when i try to read data from spark.readsttream and tr...

Data Engineering
Reply
  • 3101 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

You can just use display(orders_df3) for debugging purposes

  • 2 kudos
2 More Replies
palzor
by • New Contributor III
  • 10314 Views
  • 4 replies
  • 4 kudos

Getting error when using CDC in delta live table

Hi,I am trying to use CDC for delta live table, and when when I run the pipeline second time I get an error :org.apache.spark.sql.streaming.StreamingQueryException: Query tbl_cdc [id = ***-xx-xx-bf7e-6cb8b0deb690, runId = ***-xxxx-4031-ba74-b4b22be05...

Data Engineering
Reply
  • 10314 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 4 kudos

Hi @Palzor Lama​,A streaming live table can only process append queries; that is, queries where new rows are inserted into the source table. Processing updates from source tables, for example, merges and deletes, is not supported. To process updates,...

  • 4 kudos
3 More Replies
JeromeB974
by • New Contributor II
  • 7629 Views
  • 5 replies
  • 6 kudos

can we use spark-xml with delta live tables ?

Hiis there a way to use spark-xml with delta live tables (Azure Databricks) ?i 've try something like this without any succes for the momentCREATE LIVE TABLE df17 USING com.databricks.spark.xmlAS SELECT * FROM cloud_files("/mnt/dev/bronze/xml/s432799...

Data Engineering
Reply
  • 7629 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Zachary_Higgins
Contributor
  • 6 kudos

This is a tough one since the only magic command available is %pip, but spark-xml is a maven package. The only way I found to do this was to install the spark-xml jar from the maven repo using the databricks-cli. You can reference the cluster ID usin...

  • 6 kudos
4 More Replies
Taha_Hussain
by • Databricks Employee
  • 1105 Views
  • 0 replies
  • 1 kudos

Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next events are scheduled for ...

Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next events are scheduled for June 8th & June 22 from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts...

Data Engineering
Reply
  • 1105 Views
  • 0 replies
  • 1 kudos
thaipham
by • New Contributor III
  • 2287 Views
  • 3 replies
  • 4 kudos

Resolved! How would I export the latest revision of a notebook?

I've been trying to export some notebooks from my Databricks workspace to my laptop. I can't use Git Repos because the company restricted access to external services from the control plane.However it looks to me that I always exported the previous re...

Data Engineering
Reply
  • 2287 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

Too bad you are not allowed to use Repos, can be a life saver.Can you check your answer as best answer so the question is marked as solved?

  • 4 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels