cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

DamienSicard
by New Contributor III
  • 10550 Views
  • 2 replies
  • 1 kudos

Resolved! Notebooks font size

Hi,Is there a way to increase the cells' font size and set it as a default setting ?Thanks.Best Damien

  • 10550 Views
  • 2 replies
  • 1 kudos
Latest Reply
DamienSicard
New Contributor III
  • 1 kudos

@Werner Stinckens​ Alright, thanks for your answer.Best Damien

  • 1 kudos
1 More Replies
herry
by New Contributor III
  • 7426 Views
  • 5 replies
  • 1 kudos

Hive Serde table vs Delta table

This might be stupid question. Does the Hive Serde table have the same features (e.g. transactions) comparing to the Delta table?I tried to find the information in the Databricks documentation but I cannot find a clear answer.I create the Hive Serde ...

  • 7426 Views
  • 5 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

AFAIK Hive SerDe is just Serializer and Deserializer (write and read data to/from storage).Hive uses SerDe (and FileFormat) to read and write table rows. So it is not an actual file format like parquet, orc and also delta lake (which I consider a se...

  • 1 kudos
4 More Replies
aldrich_ang
by New Contributor II
  • 11495 Views
  • 3 replies
  • 4 kudos

Resolved! How do you use cloud fetch?

We're trying to pull a big amount of data using databricks sql and seem to have a bottleneck on network throughput when fetching the data.I see there's a new feature called cloud fetch and this seems to be the perfect solution for our issue. But I do...

  • 11495 Views
  • 3 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

Trying to get an idea of what you are trying:so you query directly on a database of +100GB or is it parquet/delta source?Also, where is the result fetched to? File download, BI tool, ...?

  • 4 kudos
2 More Replies
Constantine
by Contributor III
  • 2296 Views
  • 1 replies
  • 2 kudos

Do we have delta table access logs ?

I have delta tables on databricks with AWS s3.Are there any logs or anything else to figure out who all are accessing a particular DB or tables.

  • 2296 Views
  • 1 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

The thing that comes closest are Audit logs. Here is a list of log triggers.

  • 2 kudos
RantoB
by Valued Contributor
  • 11620 Views
  • 17 replies
  • 7 kudos

Resolved! unzip twice the same file not executing

Hi, I need to unzip some files that are ingested but when I unzip twice the same zipped file, the unzip command does not execute :As suggesgted in the documentation I did :import urllib urllib.request.urlretrieve("https://resources.lendingclub.com/L...

  • 11620 Views
  • 17 replies
  • 7 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 7 kudos

Another problem is that dbfs storage doesn't support random writes (used by zip):Does not support random writes. For workloads that require random writes, perform the operations on local disk first and then copy the result to/dbfssource: https://docs...

  • 7 kudos
16 More Replies
-werners-
by Esteemed Contributor III
  • 5472 Views
  • 0 replies
  • 19 kudos

www.lunasec.io

Log4J: part 2Apparently log4j 2.15 is still vulnerable:https://www.lunasec.io/docs/blog/log4j-zero-day-update-on-cve-2021-45046/So beter use version 2.16.But as mentioned in several topics: Databricks does not use an impacted version.

  • 5472 Views
  • 0 replies
  • 19 kudos
steven_vcnt
by New Contributor III
  • 7770 Views
  • 3 replies
  • 4 kudos

Resolved! ADLSGen2: Cannot display Delta table summary from Data>Table tab on Databricks

Hello, I have set up my account storage on Azure with an ADLSGen2 and I have succeeded to save the delta table on my ADLSGen2, from there I have created my delta table on Databricks.From there I am unable to display the summary of my delta table unde...

Errorimage
  • 7770 Views
  • 3 replies
  • 4 kudos
Latest Reply
steven_vcnt
New Contributor III
  • 4 kudos

Hello,Following Hubert comment, in order to create a delta table on Databricks from Azure, I had to use CLONE argument in order to copy the data plus the metadata of my delta table on Azure. In order to set up the connection between Databricks and A...

  • 4 kudos
2 More Replies
Redkite
by New Contributor III
  • 4655 Views
  • 7 replies
  • 2 kudos

Create table for non-admins (Table Access Control cluster/workspace)

Is there a way for non admin (at workspace level) or users without having (SELECT, MODIFY on ANY File) to create tables (unmanaged/external) even though they are owner of the database in which they want to create tables in a Table Access Controlled c...

  • 4655 Views
  • 7 replies
  • 2 kudos
Latest Reply
Blackwell15
New Contributor II
  • 2 kudos

Grant privileges on all the explain tables to non admin user as ... where BIADMIN is the non admin user who wants to generate explain plans. AdvancedMD Login

  • 2 kudos
6 More Replies
ssm3819
by New Contributor III
  • 9031 Views
  • 3 replies
  • 4 kudos

Resolved! Gstreamer installation in databricks

Hi All,i am trying to install the Gstreamer using the instructions in the below link.https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-codec-compressed-audio-input-streams?tabs=debian&pivots=programming-language-pyth...

image.png image.png
  • 9031 Views
  • 3 replies
  • 4 kudos
Latest Reply
ssm3819
New Contributor III
  • 4 kudos

Thanks you all.i have resolved this issue by running the below command.Update Packagessudo apt-get update sudo apt-get upgradesudo apt-get build-essentialsudo apt-get pkg-configsudo apt install libgstreamer1.0-0 \gstreamer1.0-plugins-base \gstreamer1...

  • 4 kudos
2 More Replies
Srikanth_Gupta_
by Databricks Employee
  • 7036 Views
  • 2 replies
  • 0 kudos

How to process images and video through structured streaming using Delta Lake?

Can we scan though videos and identify and alert in real time if something goes wrong? what are best practices for this kind of use case?

  • 7036 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Maybe I'm a little off topic, but can you recommend companies that are engaged in video production? I want to make an explanatory video for my site.

  • 0 kudos
1 More Replies
Loki
by New Contributor III
  • 10767 Views
  • 10 replies
  • 3 kudos

Apache Log4J Vulnerability

Hi Community, We got an email from our IT Team regarding Apache Log4J Vulnerability. Just wanted to understand if our implementation will be affected by this or not. We are using the following library or package in our notebooksimport org.apache.log4...

  • 10767 Views
  • 10 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

On most databricks distributions log4j version is 1.2.17

  • 3 kudos
9 More Replies
Mariusz_Cyp
by New Contributor II
  • 9796 Views
  • 3 replies
  • 12 kudos

When the billing time starts for the cluster?

Hi All, I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?​Thanks in advance! MC

  • 9796 Views
  • 3 replies
  • 12 kudos
Latest Reply
franco_patano
Databricks Employee
  • 12 kudos

Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.

  • 12 kudos
2 More Replies
Soma
by Valued Contributor
  • 4385 Views
  • 2 replies
  • 1 kudos

Resolved! AutoLoader with Custom Queue

Hi Everyone can someone help with creating custom queue for auto loader as given here as default FlushwithClose event is not getting created when my data is uploaded to blob as given in azure DB docscloudFiles.queueNameThe name of the Azure queue. If...

  • 4385 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

you need to setup notification service for blob/adls like here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#cloud-resource-managementsetUpNotificationServices will return queue name which later can be used in au...

  • 1 kudos
1 More Replies
mrvi2310
by New Contributor II
  • 8535 Views
  • 4 replies
  • 3 kudos

what is the difference between weekday and dayofweek function in spark SQL?

dayofweek: https://docs.databricks.com/sql/language-manual/functions/dayofweek.htmlweekday : https://docs.databricks.com/sql/language-manual/functions/weekday.htmlAccording to the documentation , they both are synonym functions. But when I use it I n...

weekday vs dayofweek
  • 8535 Views
  • 4 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

That's correct for weekday moday=0 for dayofweek Sunday=1.​​You can also look for documentation here https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.dayofweek.html​https://spark.apache.org/docs/latest/api/sql/index...

  • 3 kudos
3 More Replies
herry
by New Contributor III
  • 5611 Views
  • 6 replies
  • 4 kudos

CVE-2021-44228

Hi,Any affect of CVE-2021-44228 problem on Databricks platform?Is there any action that needs to be done by Databricks customer related to CVE-2021-44228?

  • 5611 Views
  • 6 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

On most databricks distributions log4j version is 1.2.17

  • 4 kudos
5 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels