- 12290 Views
- 2 replies
- 3 kudos
Please let me know how i can install PyAudio using the Databricks notebook
Hi,i am trying to install the PyAudio package.but i am getting the following error. Collecting pyaudio Using cached PyAudio-0.2.11.tar.gz (37 kB)Building wheels for collected packages: pyaudio Building wheel for pyaudio (setup.py) ... error ERROR: Co...
- 12290 Views
- 2 replies
- 3 kudos
- 3 kudos
looks like missing dependencies on the server (linux): portaudioThis should be installed:https://stackoverflow.com/questions/48690984/portaudio-h-no-such-file-or-directory
- 3 kudos
- 3774 Views
- 1 replies
- 0 kudos
Set tags for an MLFlow Experiment using Python?
There is this rest API: https://www.mlflow.org/docs/latest/rest-api.html#set-experiment-tagCan I do the same from python's MLFlow API?
- 3774 Views
- 1 replies
- 0 kudos
- 0 kudos
Someone answered first in StackOverflow. Here it is:from mlflow.tracking import MlflowClient # Create an experiment with a name that is unique and case sensitive. client = MlflowClient() experiment_id = client.create_experiment("Social NLP Experime...
- 0 kudos
- 4254 Views
- 2 replies
- 4 kudos
Resolved! Why isn't my notebook search function working?
My search function is broken. I can't search for notebook contents.
- 4254 Views
- 2 replies
- 4 kudos
- 4 kudos
Here is a tool availableelsevierlabs-os/NotebookDiscovery: Notebook Discovery Tool for Databricks notebooks (github.com)How to Catalog and Discover Your Databricks Notebooks Faster - The Databricks Blog
- 4 kudos
- 19320 Views
- 13 replies
- 3 kudos
Event hub streaming improve processing rate
Hi all,I'm working with event hubs and data bricks to process and enrich data in real-time.Doing a "simple" test, I'm getting some weird values (input rate vs processing rate) and I think I'm losing data:If you can see, there is a peak with 5k record...
- 19320 Views
- 13 replies
- 3 kudos
- 3 kudos
hi @Jhonatan Reyes​ ,How many Event hubs partitions are you readying from? your micro-batch takes a few milliseconds to complete, which I think is good time, but I would like to undertand better what are you trying to improve here.Also, in this case ...
- 3 kudos
- 8362 Views
- 5 replies
- 5 kudos
Capture num_affected_rows in notebooks
If I run some code, say for an ETL process to migrate data from bronze to silver storage, when a cell executes it reports num_affected_rows in a table format. I want to capture that and log it in my logger. Is it stored in a variable or syslogged som...
- 8362 Views
- 5 replies
- 5 kudos
- 5 kudos
Good one Dan! I never thought of using the delta api for this but there you go.
- 5 kudos
- 2093 Views
- 1 replies
- 1 kudos
- 2093 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @xiaojun wang​ please check the blog and let us know if this helps you.https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html
- 1 kudos
- 9907 Views
- 8 replies
- 7 kudos
How to optimize exporting dataframe to delta file?
Scenario : I have a dataframe that have 5 billion records/rows and 100+ columns. Is there a way to write this in a delta format efficiently. I have tried to export it but cancelled it after 2 hours (write didnt finish) as this processing time is not ...
- 9907 Views
- 8 replies
- 7 kudos
- 7 kudos
Hi @Franco Sia​ ,I will recommend to avoid to use the repartition(50), instead enable optimizes writes on your Delta table. You can find more details hereEnable optimized writes and auto compaction on your Delta table. Use AQE (docs here) to have eno...
- 7 kudos
- 11531 Views
- 10 replies
- 6 kudos
Older Spark Version loaded into the spark notebook
I have databricks runtime for a job set to latest 10.0 Beta (includes Apache Spark 3.2.0, Scala 2.12) .In the notebook when I check for the spark version, I see version 3.1.0 instead of version 3.2.0I need the Spark version 3.2 to process workloads a...
- 11531 Views
- 10 replies
- 6 kudos
- 6 kudos
hi @Dhaivat Upadhyay​ ,Good news, DBR 10 was release yesterday October 20th. You can find more details in the release notes website
- 6 kudos
- 2763 Views
- 2 replies
- 6 kudos
Where can I enable notification for comments made on my notebook?
I'm not being communicated when a comment is made on a notebook, is it possible to enable?
- 2763 Views
- 2 replies
- 6 kudos
- 26382 Views
- 15 replies
- 6 kudos
Resolved! BufferHolder Exceeded on Json flattening
On Databricks, we use the following code to flatten JSON in Python. The data is from a REST API:```df = spark.read.format("json").option("header", "true").option("multiline", "true").load(SourceFileFolder + sourcetable + "*.json")df2 = df.select(psf....
- 26382 Views
- 15 replies
- 6 kudos
- 6 kudos
@Dennis D​ , what's happening here is that more than 2 GB (2147483648 bytes) is being loaded into a single column value. This is a hard-limit for serialization. This KB article addresses it. The solution would be to find some way to have this loaded ...
- 6 kudos
- 2846 Views
- 4 replies
- 3 kudos
Feature request: It is possible to add comments to both databricks sql databases and tables. It would be really usefull if these comments could show u...
Feature request: It is possible to add comments to both databricks sql databases and tables. It would be really usefull if these comments could show up (if they are provided) in PowerBI when one connects to the Databricks SQL endpoint, e.g. in this w...
- 2846 Views
- 4 replies
- 3 kudos
- 5590 Views
- 6 replies
- 5 kudos
Resolved! How to implement the where not exists pattern in scala?
I have a dataframe with the following columns:Key1Key2Y_N_ColCol1Col2For the key tuple (Key1, Key2), I have rows with Y_N_Col = "Y" and Y_N_Col = "N".I need a new dataframe with all rows with Y_N_Col = "Y" (regardless of the key tuple), plus all Y_N_...
- 5590 Views
- 6 replies
- 5 kudos
- 5 kudos
I'd use a left-anti join.So create a df with all the Y, then create a df with all the N and do a left_anti join (on key1 and key2) on the df with the Y.then a union of those two.
- 5 kudos
- 1018 Views
- 0 replies
- 0 kudos
aws logo
What is the future of aws?The future of AWS is very promising. So, if you are thinking of a cloud career or want to switch your position to something related to the cloud, I would highly recommend you going for AWS training. No matter what field you ...
- 1018 Views
- 0 replies
- 0 kudos
- 2817 Views
- 0 replies
- 0 kudos
- 2817 Views
- 0 replies
- 0 kudos
- 2405 Views
- 1 replies
- 0 kudos
If data from a Delta table is cached in Databricks SQL and the table is altered in the backend, does it invalidate the cache?
Basically I'm worried about the scenario where data that gets cached on Databricks SQL endpoints becomes out of sync with the source Delta table. If that were to happen and data was read from the cache it would be out of date/incorrect. Is this a con...
- 2405 Views
- 1 replies
- 0 kudos
- 0 kudos
There are 3 types of caching. 1-Databricks SQL UI caching, 2-Query results caching , 3-Delta caching . (1) does not get invalidated. It's like your BI dashboard. BI dashboard needs to be manually refreshed.(2) and (3) gets auto invalidation.pls check...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
