cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Srikanth_Gupta_
by Valued Contributor
  • 3467 Views
  • 2 replies
  • 0 kudos

How to process images and video through structured streaming using Delta Lake?

Can we scan though videos and identify and alert in real time if something goes wrong? what are best practices for this kind of use case?

  • 3467 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Maybe I'm a little off topic, but can you recommend companies that are engaged in video production? I want to make an explanatory video for my site.

  • 0 kudos
1 More Replies
Loki
by New Contributor III
  • 6506 Views
  • 11 replies
  • 3 kudos

Apache Log4J Vulnerability

Hi Community, We got an email from our IT Team regarding Apache Log4J Vulnerability. Just wanted to understand if our implementation will be affected by this or not. We are using the following library or package in our notebooksimport org.apache.log4...

  • 6506 Views
  • 11 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Lokesh Sharma​ , Please go through the blog for further information on the Apache Log4J Vulnerability issue.

  • 3 kudos
10 More Replies
herry
by New Contributor III
  • 3571 Views
  • 8 replies
  • 5 kudos

Resolved! CVE-2021-44228

Hi,Any affect of CVE-2021-44228 problem on Databricks platform?Is there any action that needs to be done by Databricks customer related to CVE-2021-44228?

  • 3571 Views
  • 8 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Herry Ramli​ , Please go through the blog for more information regarding the issue stated by you.

  • 5 kudos
7 More Replies
Mariusz_Cyp
by New Contributor II
  • 5220 Views
  • 3 replies
  • 7 kudos

When the billing time starts for the cluster?

Hi All, I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?​Thanks in advance! MC

  • 5220 Views
  • 3 replies
  • 7 kudos
Latest Reply
franco_patano
New Contributor III
  • 7 kudos

Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.

  • 7 kudos
2 More Replies
mrvi2310
by New Contributor II
  • 4340 Views
  • 7 replies
  • 3 kudos

what is the difference between weekday and dayofweek function in spark SQL?

dayofweek: https://docs.databricks.com/sql/language-manual/functions/dayofweek.htmlweekday : https://docs.databricks.com/sql/language-manual/functions/weekday.htmlAccording to the documentation , they both are synonym functions. But when I use it I n...

weekday vs dayofweek
  • 4340 Views
  • 7 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @mrvi2310​ , As @Hubert Dudek​ has rightly mentioned, here are the screenshots:-References:- https://spark.apache.org/docs/latest/api/sql/

  • 3 kudos
6 More Replies
Mohit_m
by Valued Contributor II
  • 1913 Views
  • 1 replies
  • 5 kudos

How to find out the users who accessed Databricks and from which location

How to find out the users who accessed Databricks and from which location

  • 1913 Views
  • 1 replies
  • 5 kudos
Latest Reply
Mohit_m
Valued Contributor II
  • 5 kudos

You can use Audit logs to fetch this dataQuery:%sqlSELECT DISTINCT userIdentity.email, sourceIPAddressFROM audit_logsWHERE serviceName = "accounts" AND actionName LIKE "%login%"Please find below the docs to analyse the Audit logshttps://docs.databric...

  • 5 kudos
JonHMDavis
by New Contributor II
  • 3267 Views
  • 5 replies
  • 4 kudos

Resolved! Graphframes not importing on Databricks 9.1 LTS ML

Is Graphframes for python meant to be installed by default on Databricks 9.1 LTS ML? Previously I was running the attached python command on 7.3 LTS ML with no issue, however now I am getting "no module named graphframes" when trying to import the pa...

  • 3267 Views
  • 5 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

Hm I thought the same thing as the docs state "The GraphFrames package is included in Databricks Runtime for Machine Learning".But that could have changed...

  • 4 kudos
4 More Replies
-werners-
by Esteemed Contributor III
  • 2253 Views
  • 3 replies
  • 14 kudos

Notebook fails in job but not in interactive mode

I have this notebook which is scheduled by Data Factory on a daily basis.It works fine, up to today. All of a sudden I keep on getting NullpointerException when writing the data.After some searching online, I disabled AQE. But this does not help.Th...

  • 2253 Views
  • 3 replies
  • 14 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 14 kudos

After some tests it seems that if I run the notebook on an interactive cluster, I only get 80% of load (Ganglia metrics).If I run the same notebook on a job cluster with the same VM types etc (so the only difference is interactive vs job), I get over...

  • 14 kudos
2 More Replies
RiyazAli
by Valued Contributor
  • 8347 Views
  • 9 replies
  • 5 kudos

Resolved! Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

I'm trying to read a small txt file which is added as a table to the default db on Databricks. While trying to read the file via Local File API, I get a `FileNotFoundError`, but I'm able to read the same file as Spark RDD using SparkContext.Please fi...

  • 8347 Views
  • 9 replies
  • 5 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

can you try with /dbfs/Filestore/tables/boringwords.txt?

  • 5 kudos
8 More Replies
pjp94
by Contributor
  • 1533 Views
  • 4 replies
  • 9 kudos

Databrick Job - Notebook Execution

Question - When you set a reoccuring job to simply update a notebook, does databricks clear the state of the notebook prior to executing the notebook? If not, can I configure it to make sure it clears the state before running?

  • 1533 Views
  • 4 replies
  • 9 kudos
Latest Reply
Anonymous
Not applicable
  • 9 kudos

@Paras Patel​ - Would you be happy to mark Hubert's answer as best so that other members can find the solution more easily?Thanks!

  • 9 kudos
3 More Replies
morganmazouchi
by New Contributor III
  • 5431 Views
  • 7 replies
  • 2 kudos

Resolved! Incremental updates in Delta Live Tables

What happens if we change the logic for the delta live tables and we do an incremental update. Does the table get reset (refresh) automatically or would it only apply the logic to new incoming data? would we have to trigger a reset in this case?

  • 5431 Views
  • 7 replies
  • 2 kudos
Latest Reply
morganmazouchi
New Contributor III
  • 2 kudos

Here is my finding on when to refresh (reset) the table: If it is a complete table all the changes would be apply automatically. If the table is incremental table, you need to do a manually reset (full refresh).

  • 2 kudos
6 More Replies
Kody_Devl
by New Contributor II
  • 3434 Views
  • 3 replies
  • 2 kudos

%SQL Append null values into a SQL Table

Hi All, I am new to Databricks and am writing my first program.Note: Code Shown Below:I am creating a table with 3 columns to store data. 2 of the columns will be appended in from data that I have in another table.When I run my append query into the...

  • 3434 Views
  • 3 replies
  • 2 kudos
Latest Reply
Kody_Devl
New Contributor II
  • 2 kudos

Hi Hubert,Your answer moves me closer to being able to update pieces of a 26 field MMR_Restated table in pieces are the correct fields values are calculated Thru the process. I have been looking for a way to be able to update in "pieces"...... 2 fie...

  • 2 kudos
2 More Replies
ak09
by New Contributor
  • 615 Views
  • 0 replies
  • 0 kudos

Triggering Notebook in Azure Repos via Azure DevOps

I have been using Databricks workspace for all my data science projects in my firm. In my current project, I have built a CI pipeline using databricks-cli & Azure DevOps. Using databricks-cli I can trigger the Notebook which is present in my workspa...

  • 615 Views
  • 0 replies
  • 0 kudos
tarente
by New Contributor III
  • 2620 Views
  • 3 replies
  • 3 kudos

Partitioned parquet table (folder) with different structure

Hi,We have a parquet table (folder) in Azure Storage Account.The table is partitioned by column PeriodId (represents a day in the format YYYYMMDD) and has data from 20181001 until 20211121 (yesterday).We have a new development that adds a new column ...

  • 2620 Views
  • 3 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

I think problem is in overwrite as when you overwrite it overwrites all folders. Solution is to mix append with dynamic overwrite so it will overwrite only folders which have data and doesn't affect old partitions:spark.conf.set("spark.sql.sources.pa...

  • 3 kudos
2 More Replies
SQL
by New Contributor II
  • 1688 Views
  • 5 replies
  • 1 kudos

Presto hive table to delta table conversion

Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with  ex...

  • 1688 Views
  • 5 replies
  • 1 kudos
Latest Reply
SQL
New Contributor II
  • 1 kudos

could you please help me with the alternative function for date_array

  • 1 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels