- 10960 Views
- 5 replies
- 2 kudos
Resolved! Call Databricks notebook in a specific branch from Azure Data Factory?
I'm using the new Databricks Repos functionality and in Azure Data Factory UI for the notebook activity you can browse the Databricks workspace and select Repos > username > project > folder > notebook.Is it possible to call a Databricks notebook in ...
- 10960 Views
- 5 replies
- 2 kudos
- 2 kudos
Greetings, I have similar problem. Did you try to use Databricks workflows instead and schedule them instead on Data Factory?Because inside workflows it is possible to select a specific branch, so it may actually work.What do you think?
- 2 kudos
- 2530 Views
- 2 replies
- 2 kudos
Data explorer missing a vital scroller
hinot sure best way to raise an issue, working in Azure DB1)the horizonal scroll bar has gone walkabouts and is now at the end of the vertical (in Chrome, was ok a few weeks ago) (see image)can we have it back to the bottom of the screen please?also...
- 2530 Views
- 2 replies
- 2 kudos
- 2 kudos
it is not a question really , more a request to fix the bug. scrollers not working as they should
- 2 kudos
- 1135 Views
- 1 replies
- 1 kudos
Markdown issues with adding text in math mode
I am trying to use the \text function from the package amsmath, but I can't seem to find a way to call this package. Is there another route of achieving the same result? Thanks!
- 1135 Views
- 1 replies
- 1 kudos
- 1 kudos
can you share exact code by that we can also test the same and help you in better manner
- 1 kudos
- 2275 Views
- 1 replies
- 0 kudos
How can I create grouped bars that are also stacked visualization?
I wish to create a visualization combined of grouped bars, and also have those bars stacked.Attached is a sketch of the final result I am interested in.I am also attaching my sql because I'm not sure if I should "group by" in the query or in the visu...
- 2275 Views
- 1 replies
- 0 kudos
- 0 kudos
try to use asc or dsc keyword ,maybe it worked
- 0 kudos
- 1613 Views
- 1 replies
- 0 kudos
JDBC to delta lake: Is setting fetch size expected to be effective?
I am using the databricks jdbc driver to access a delta lake. The database URL specifies transportMode=http. I have experimented with setting different values of fetchSize on the java.sqlPreparedStatement object and have monitored memory use within m...
- 1613 Views
- 1 replies
- 0 kudos
- 0 kudos
I think there is one spark configuration but I forgot right now Pelase try to utilized this doc maybe you get something- https://spark.apache.org/docs/latest/configuration.html
- 0 kudos
- 2202 Views
- 1 replies
- 0 kudos
Access same createorreplacetempview("viewname") by multiple functions.
I have several functions accessing the same createorreplacetempview("viewname"). Does this cause any issues with multiple functions accessing it in a distributed environment?def get_data_sql(spark_session, data_frame, data_element): data_fram...
- 2202 Views
- 1 replies
- 0 kudos
- 0 kudos
there is two type of viewsone is global view - it will be available for whole cluster and notebook but it will removed after cluster restartand another is Temp view- that will be available for only notebook level, and other notebook will not able to ...
- 0 kudos
- 1036 Views
- 1 replies
- 0 kudos
With AWS/Azure Autoscaling, how do we fine tune spark jobs?
With the recommended autoscaling, e.g, https://docs.databricks.com/clusters/cluster-config-best-practices.html, setting; is it possible to dynamically set a fine tuned spark job, given that the number of executors could be changing at any time?
- 1036 Views
- 1 replies
- 0 kudos
- 0 kudos
@Andrew Fogarty I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost
- 0 kudos
- 5097 Views
- 1 replies
- 1 kudos
How to read a PDF file from Azure Datalake blob storage to Databricks
I have a scenario where I need to read a pdf file from "Azure Datalake blob storage to Databricks", where connection is done through AD access.Generating the SAS token has been restricted in our environment due to security issues. The below script ca...
- 5097 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Punith raj ,Not sure about Azure but in AWS there is one service known as AWS Transact Please try to explore that onces
- 1 kudos
- 6918 Views
- 6 replies
- 7 kudos
Resolved! Wrong X-Axis Order when Visualization is Put on Dashboard
I have a visualization in which the X-axis values are displayed correctly in the Query Editor, in the order produced by the SQL query. However, when I add the visualization to a dashboard, the values are suddenly not sorted anymore.How is this possib...
- 6918 Views
- 6 replies
- 7 kudos
- 7 kudos
We have further analyzed the visualization problem and found two solutions.The original visualization consists of 1 series and has aggregation enabled in the UI (but is unused, since the query itself aggregates already).We found that the following tw...
- 7 kudos
- 5901 Views
- 2 replies
- 0 kudos
Resolved! Best way to get one row back per ID? Select Distinct is not working.
Here is the current output for my select statement. I would like it to return one row for this jobsubmissionid, where it selects only the non-zero value from each of the rows. I tried using SELECT DISTINCT jobsubmissionidbut it still returned 5 rows.
- 5901 Views
- 2 replies
- 0 kudos
- 0 kudos
Is that the complete query you are using. I'm guessing that you are using select distinct * from table_name. If you wanted a individual column distinct value you have to apply a filter condition or aggregate the data accordingly. Anyways, a complete ...
- 0 kudos
- 17944 Views
- 11 replies
- 52 kudos
Resolved! Data Engineering with Databricks V2 or V3 for qualification in a few months time
If I am new to Databricks and is aiming to get qualification some point Dec2022 or Jan 2023, should I be studying the material Data Engineering with Databricks V2 or V3?
- 17944 Views
- 11 replies
- 52 kudos
- 52 kudos
I would suggest to go for V3 because the course Data Engineering with Databricks (V3) is the latest version as of now and was released on 14th October 2022. So, this version would have more topics in comparison to V2.
- 52 kudos
- 26215 Views
- 19 replies
- 42 kudos
Resolved! Data Engineering Professional Practice exam
I'd like to ask if there is a tentative date to release Databricks Data Engineering practice exam. Thank you!
- 26215 Views
- 19 replies
- 42 kudos
- 42 kudos
No, as of now there is no practice exam available for this certification but a good way to get an idea about the exam would be appearing for it once. There are multiple trainings going on from Databricks, attending which you can get the voucher code ...
- 42 kudos
- 1442 Views
- 0 replies
- 0 kudos
UDF function while registering- PicklingError
PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers.I...
- 1442 Views
- 0 replies
- 0 kudos
- 7812 Views
- 6 replies
- 2 kudos
Resolved! How to see if condition is True / False for all rows in a DataFrame?
Assume that I have a Spark DataFrame, and I want to see if records satisfy a condition.Example dataset:# Prepare Data data = [('A', 1), \ ('A', 2), \ ('B', 3) ] # Create DataFrame columns= ['col_1', 'col_2'] df = spark.createDataF...
- 7812 Views
- 6 replies
- 2 kudos
- 2 kudos
Hi you can use display() or show() function that will provide you expected results.
- 2 kudos
- 1709 Views
- 1 replies
- 2 kudos
Unable to read image and vedio data in Databricks using OpenCV.
I have tried reading image and video data in Azure databricks using OpenCv. When I have checked the type of image, it’s shown as “NonType” and when I tried with vedio file, the file itself was not being opened. (Note: these files are stored on azure ...
- 1709 Views
- 1 replies
- 2 kudos
- 2 kudos
Kindly let me know if you find the answer.!!
- 2 kudos