- 13343 Views
- 4 replies
- 2 kudos
Resolved! Error when importing PyDeequ package
Hi everyone,I want to do some tests regarding data quality and for that I pretend to use PyDeequ on a databricks notebook. Keep in mind that I'm very new to databricks and Spark.First I created a cluster with the Runtime version "10.4 LTS (includes A...
- 13343 Views
- 4 replies
- 2 kudos
- 2 kudos
I assumed I wouldn't need to add the Deequ library. Apparently, all I had to do was add it via Maven coordinates and it solved the problem.
- 2 kudos
- 3582 Views
- 1 replies
- 0 kudos
Jupyter notebooks import in databricks notebooks
Dear Team,Is it possible to import jupyter notebooks in databricks community edition ? if yes will there be any formatting issues ?
- 3582 Views
- 1 replies
- 0 kudos
- 0 kudos
if yes is there any limit ? , what is the difference or advantage using juypter notebooks over databricks notebooks
- 0 kudos
- 2635 Views
- 2 replies
- 0 kudos
What is the underlying database used in data bricks community edition in sql
Dear DB Experts,I am reaching out to check whether can i still use postgressql in notebooks with notebook as sql and try postgresql , as far as i know from reading the back end db is mysql correct me my understanding ?
- 2635 Views
- 2 replies
- 0 kudos
- 1002 Views
- 0 replies
- 0 kudos
Why does the Spark user differ if using sparklyr from Spark SQL?
My team uses a shared cluster. We've been having issues with spark_connect failing to work at times (can't easily reproduce). One thing I've recently noticed is that the Spark user through sparklyr seems to be set to the first person who connects to ...
- 1002 Views
- 0 replies
- 0 kudos
- 16156 Views
- 5 replies
- 2 kudos
Resolved! Call Databricks notebook in a specific branch from Azure Data Factory?
I'm using the new Databricks Repos functionality and in Azure Data Factory UI for the notebook activity you can browse the Databricks workspace and select Repos > username > project > folder > notebook.Is it possible to call a Databricks notebook in ...
- 16156 Views
- 5 replies
- 2 kudos
- 2 kudos
Greetings, I have similar problem. Did you try to use Databricks workflows instead and schedule them instead on Data Factory?Because inside workflows it is possible to select a specific branch, so it may actually work.What do you think?
- 2 kudos
- 4533 Views
- 2 replies
- 2 kudos
Data explorer missing a vital scroller
hinot sure best way to raise an issue, working in Azure DB1)the horizonal scroll bar has gone walkabouts and is now at the end of the vertical (in Chrome, was ok a few weeks ago) (see image)can we have it back to the bottom of the screen please?also...
- 4533 Views
- 2 replies
- 2 kudos
- 2 kudos
it is not a question really , more a request to fix the bug. scrollers not working as they should
- 2 kudos
- 2336 Views
- 1 replies
- 1 kudos
Markdown issues with adding text in math mode
I am trying to use the \text function from the package amsmath, but I can't seem to find a way to call this package. Is there another route of achieving the same result? Thanks!
- 2336 Views
- 1 replies
- 1 kudos
- 1 kudos
can you share exact code by that we can also test the same and help you in better manner
- 1 kudos
- 6007 Views
- 1 replies
- 1 kudos
How can I create grouped bars that are also stacked visualization?
I wish to create a visualization combined of grouped bars, and also have those bars stacked.Attached is a sketch of the final result I am interested in.I am also attaching my sql because I'm not sure if I should "group by" in the query or in the visu...
- 6007 Views
- 1 replies
- 1 kudos
- 1 kudos
try to use asc or dsc keyword ,maybe it worked
- 1 kudos
- 3518 Views
- 1 replies
- 0 kudos
JDBC to delta lake: Is setting fetch size expected to be effective?
I am using the databricks jdbc driver to access a delta lake. The database URL specifies transportMode=http. I have experimented with setting different values of fetchSize on the java.sqlPreparedStatement object and have monitored memory use within m...
- 3518 Views
- 1 replies
- 0 kudos
- 0 kudos
I think there is one spark configuration but I forgot right now Pelase try to utilized this doc maybe you get something- https://spark.apache.org/docs/latest/configuration.html
- 0 kudos
- 3366 Views
- 1 replies
- 0 kudos
Access same createorreplacetempview("viewname") by multiple functions.
I have several functions accessing the same createorreplacetempview("viewname"). Does this cause any issues with multiple functions accessing it in a distributed environment?def get_data_sql(spark_session, data_frame, data_element): data_fram...
- 3366 Views
- 1 replies
- 0 kudos
- 0 kudos
there is two type of viewsone is global view - it will be available for whole cluster and notebook but it will removed after cluster restartand another is Temp view- that will be available for only notebook level, and other notebook will not able to ...
- 0 kudos
- 1915 Views
- 1 replies
- 0 kudos
With AWS/Azure Autoscaling, how do we fine tune spark jobs?
With the recommended autoscaling, e.g, https://docs.databricks.com/clusters/cluster-config-best-practices.html, setting; is it possible to dynamically set a fine tuned spark job, given that the number of executors could be changing at any time?
- 1915 Views
- 1 replies
- 0 kudos
- 0 kudos
@Andrew Fogarty​ I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost
- 0 kudos
- 14561 Views
- 6 replies
- 7 kudos
Resolved! Wrong X-Axis Order when Visualization is Put on Dashboard
I have a visualization in which the X-axis values are displayed correctly in the Query Editor, in the order produced by the SQL query. However, when I add the visualization to a dashboard, the values are suddenly not sorted anymore.How is this possib...
- 14561 Views
- 6 replies
- 7 kudos
- 7 kudos
We have further analyzed the visualization problem and found two solutions.The original visualization consists of 1 series and has aggregation enabled in the UI (but is unused, since the query itself aggregates already).We found that the following tw...
- 7 kudos
- 11494 Views
- 2 replies
- 0 kudos
Resolved! Best way to get one row back per ID? Select Distinct is not working.
Here is the current output for my select statement. I would like it to return one row for this jobsubmissionid, where it selects only the non-zero value from each of the rows. I tried using SELECT DISTINCT jobsubmissionidbut it still returned 5 rows.
- 11494 Views
- 2 replies
- 0 kudos
- 0 kudos
Is that the complete query you are using. I'm guessing that you are using select distinct * from table_name. If you wanted a individual column distinct value you have to apply a filter condition or aggregate the data accordingly. Anyways, a complete ...
- 0 kudos
- 23591 Views
- 11 replies
- 52 kudos
Resolved! Data Engineering with Databricks V2 or V3 for qualification in a few months time
If I am new to Databricks and is aiming to get qualification some point Dec2022 or Jan 2023, should I be studying the material Data Engineering with Databricks V2 or V3?
- 23591 Views
- 11 replies
- 52 kudos
- 52 kudos
I would suggest to go for V3 because the course Data Engineering with Databricks (V3) is the latest version as of now and was released on 14th October 2022. So, this version would have more topics in comparison to V2.
- 52 kudos
- 39604 Views
- 19 replies
- 44 kudos
Resolved! Data Engineering Professional Practice exam
I'd like to ask if there is a tentative date to release Databricks Data Engineering practice exam. Thank you!
- 39604 Views
- 19 replies
- 44 kudos
- 44 kudos
No, as of now there is no practice exam available for this certification but a good way to get an idea about the exam would be appearing for it once. There are multiple trainings going on from Databricks, attending which you can get the voucher code ...
- 44 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
