cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Charmin
by New Contributor
  • 561 Views
  • 1 replies
  • 0 kudos

Why 'runCommand' action does NOT show up in databricksNotebook audit log table?

I understand databricks can send diagnostic/audit logs to log analytics in azure. There is a standard 'DatabricksNotebook' table that provides audit log for notebook actions. In this table there is an action called 'runCommand' but this does not show...

  • 561 Views
  • 1 replies
  • 0 kudos
Latest Reply
rsamant07
New Contributor III
  • 0 kudos

Hi @Charmin patel​  , you need to enable verbose audit logging in workspace setting for runCommand to appear in the audit logs

  • 0 kudos
laksh
by New Contributor II
  • 811 Views
  • 2 replies
  • 0 kudos

Real time data quality validation (Streaming data ingestion)

I was wondering how the Unity Catalog would help in data quality validations for real time (streaming data) data ingestion. 

  • 811 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @arun laksh​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
mahesh_vardhan_
by New Contributor
  • 2019 Views
  • 2 replies
  • 2 kudos

Resolved! How do I use numpy case when condition in pyspark.pandas?

I do have some legacy pandas codes which I want to migrate to spark to leaverage parellelization in Databricks. I see datadricks has launched a wrapper package on top of pandas which uses pandas nomenclature but use spark engine in the backend.I comf...

  • 2019 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @mahesh vardhan gandhi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 2 kudos
1 More Replies
Paras
by New Contributor II
  • 1224 Views
  • 4 replies
  • 6 kudos
  • 1224 Views
  • 4 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Paras Gadhiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 6 kudos
3 More Replies
JustinDM
by New Contributor II
  • 654 Views
  • 2 replies
  • 2 kudos

I have been getting a False Alert on Data bricks SQL.

When ever a legitimate alert is triggered, I get a False Alert with 0.00 triggered at 12.00AM the next day. I tried Altering the Query but its still the same. Not posting examples as data is not shareable but I can give an example. if the alert is se...

  • 654 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @justin moorthy​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
Mustafa91
by New Contributor II
  • 634 Views
  • 2 replies
  • 3 kudos

Databricks job api validation

Hey everyone,​I am creating databricks Jobs using ADO pipelines. I am creating the json content using python and in thr release pipeline i call databricks cli create command with the json. ​W​hat I would like to do is that in my CI pipeline, I need t...

  • 634 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Mustafa Akilli​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedba...

  • 3 kudos
1 More Replies
kkawka1
by New Contributor III
  • 1495 Views
  • 2 replies
  • 1 kudos

Resolved! Data explorer in the community edition

Hi,Does anyone know how to access data explorer in the community edition? I would like to have an overview of what files are saved in the FileStore. This is what happens when I select "Data" in the left-hand side menu

image
  • 1495 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Konrad Kawka​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 1 kudos
1 More Replies
Sanmati
by New Contributor II
  • 911 Views
  • 2 replies
  • 2 kudos

Resolved! Request for reattempt voucher. Databricks Certified Data Engineer Associate exam

HiOn Feb 27th ,I attempted the Databricks Certified Data Engineer Associate exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 64.88%.I am planning to reattempt the exam, Could you kindly give me a...

  • 911 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Sanmati Mahesh Undodi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 2 kudos
1 More Replies
gangs
by New Contributor
  • 883 Views
  • 3 replies
  • 3 kudos

Resolved! Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table

Is any better way to load huge zipped CSV file to hive_metastore table ?????​

  • 883 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Ankit Gangwal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 3 kudos
2 More Replies
Naeem_K
by New Contributor III
  • 860 Views
  • 5 replies
  • 0 kudos

Data Engineer Associate Certificate and badge not yet received

I have cleared the certification exam on 26th January 2023, but still haven't received the certificate. I had given the exam with a different mail ID but I'm not receiving any emails from Databricks to that mail ID.​Kindly help me resolve the issue.

  • 860 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Naeemah Khatib​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
4 More Replies
Galdino
by New Contributor II
  • 2980 Views
  • 3 replies
  • 1 kudos

How to read a json from BytesIO with PySpark?

I want read a json from IO variable using PySpark.My code using pandas:io = BytesIO()ftp.retrbinary('RETR '+ file_name, io.write)io.seek(0)# With pandasdf = pd.read_json(io)What I tried using PySpark, but don't work: io = BytesIO() ftp.retrbinary('...

  • 2980 Views
  • 3 replies
  • 1 kudos
Latest Reply
Erik_L
Contributor II
  • 1 kudos

Just use pandas and follow with spark.createDataFrame(df)

  • 1 kudos
2 More Replies
iwan_aucamp
by New Contributor III
  • 820 Views
  • 2 replies
  • 2 kudos

Are there any python API bindings for the Databricks Account API?

All I could find in terms of API bindings for python is https://pypi.org/project/databricks-cli/, and this does not include the Account API and it is also not official.I will just use the OpenAPI spec, but just want to be sure I'm not doing unnecessa...

  • 820 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Iwan Aucamp​ : Yes, there are Python API bindings available for the Databricks Account API.For Databricks Account API with Python, please refer to the Databricks documentation: https://docs.databricks.com/dev-tools/api/latest/accounts.html#python-ap...

  • 2 kudos
1 More Replies
Sagar1
by New Contributor III
  • 3157 Views
  • 7 replies
  • 7 kudos

Resolved! Notebook dropdown widget

I have created a dropdown (say B) in my notebook whose input depend on dropdown( say B). So if select some value in dropdown A, it corresponding value appears in B dropdown & i'm selecting one amongst it. Now if i change the value in dropdown A, then...

  • 3157 Views
  • 7 replies
  • 7 kudos
Latest Reply
nic_paul24
New Contributor II
  • 7 kudos

If the previously selected value of B is not meant to be in the list of values for newly selected dropdown A value, then you could set a default value (ie: 'No selection') that the B dropdown should have when first created. In a method to define how ...

  • 7 kudos
6 More Replies
weldermartins
by Honored Contributor
  • 4044 Views
  • 7 replies
  • 35 kudos

Resolved! pyspark - regexp_extract

hello everyone, I'm creating a regex expression to fetch only the value of a string, but some values ​​are negative. I am not able to create the rule to compose the negative value. can you help me?from pyspark.sql.functions import regexp_extract fro...

image
  • 4044 Views
  • 7 replies
  • 35 kudos
Latest Reply
ErinArmistead
New Contributor II
  • 35 kudos

Have you found the answer? If you are a student in college or school searching for free essay examples online, you may want to visit the website https://writinguniverse.com/free-essay-examples/soccer/ here you will find a vast collection of free essa...

  • 35 kudos
6 More Replies
Baumeister
by New Contributor II
  • 1558 Views
  • 2 replies
  • 0 kudos

Error when importing .dbc of a complete Workspace

I saved the content of an older Databricks Workspace by clicking on the Dropdown next to Workspace -> Export -> DBC Archive and saved it on my local machine.In a new Databricks Workspace, I now want to import That .DBC archive to restore the previous...

dbcerror
  • 1558 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Sebastian K​ :It looks like the error you are facing while importing the DBC archive could be due to the version incompatibility between the Databricks instance where you created the DBC archive and the one where you are trying to import it. Can you...

  • 0 kudos
1 More Replies
Labels
Top Kudoed Authors